Methods and apparatus relating to a camera including multiple optical chains

ABSTRACT

Camera methods and apparatus are described where the camera device includes multiple optical chains. In various embodiments two or more of the optical chains include light redirection devices such as mirrors or prisms. Sensors corresponding to multiple different optical chains, but not necessarily all optical chains, are parallel to each other. In some embodiments sensors corresponding to different optical chains are located in the same plane at the front or rear of the camera. However other sensor mounting positions are also possible.

RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/085,961 filed on Mar. 30, 2016 which is a continuation ofU.S. patent application Ser. No. 14/516,568 filed Oct. 16, 2014 whichclaims the benefit of the following U.S. Provisional patentapplications: 61/978,818 filed Apr. 11, 2014; 61/981,849 filed Apr. 20,2014; 62/021,094 filed Jul. 4, 2014; 61/893,100 filed Oct. 18, 2013;61/896,069 filed Oct. 26, 2013; 61/899,097 filed Nov. 1, 2013;61/922,801 filed Dec. 31, 2013; 61/943,299 filed Feb. 21, 2014; and61/943,302 filed Feb. 21, 2014; and is a continuation-in-part of each ofthe following U.S. patent applications: Ser. No. 14/327,510 filed Jul.9, 2014; Ser. No. 14/327,512 filed Jul. 9, 2014; Ser. No. 14/327,514filed Jul. 9, 2014; Ser. No. 14/327,515 filed Jul. 9, 2014; Ser. No.14/327,517 filed Jul. 9, 2014; Ser. No. 14/327,518 filed Jul. 9, 2014;Ser. No. 14/327,492 filed Jul. 9, 2014; Ser. No. 14/327,524 filed Jul.9, 2014; Ser. No. 14/327,525 filed Jul. 9, 2014; and Ser. No. 14/327,508filed Jul. 9, 2014, and all of the aforementioned patent applicationsare hereby incorporated by reference in their entirety.

FIELD

The present application relates to camera methods and apparatus and,more particularly, to methods and apparatus related to camera apparatusincluding multiple optical chains, e.g., optical chains that may haveoptical paths that are longer than a camera is deep.

DISCUSSION

High quality digital cameras have to a large extent replaced filmcameras. However, like film cameras, with digital cameras much attentionhas been placed by the camera industry on the size and quality of lenseswhich are used on the camera. Individuals seeking to take qualityphotographs are often encouraged to invest in large bulky and oftencostly lenses for a variety of reasons. Among the reasons for usinglarge aperture lenses is their ability to capture a large amount oflight in a given time period as compared to smaller aperture lenses.Telephoto lenses tend to be large not only because of their largeapertures but also because of their long focal lengths. Generally, thelonger the focal length, the larger the lens. A long focal length givesthe photographer the ability to take pictures from far away.

In the quest for high quality photos, the amount of light which can becaptured is often important to the final image quality. Having a largeaperture lens allows a large amount of light to be captured allowing forshorter exposure times than would be required to capture the same amountof light using a small lens. The use of short exposure times can reduceblurriness especially with regard to images with motion. The ability tocapture large amounts of light can also facilitate the taking of qualityimages even in low light conditions. In addition, using a large aperturelens makes it possible to have artistic effects such as small depth offield for portrait photography.

Large lenses sometimes also offer the opportunity to support mechanicalzoom features that allow a user to optically zoom in or out and/or toalter the focal length of the lens which is important for framing ascene without the need to move closer or further from the subject.

While large lenses have many advantages with regard to the ability tocapture relatively large amounts of light compared to smaller lenses,support large zoom ranges, and often allow for good control over focus,there are many disadvantages to using large lenses.

Large lenses tend to be heavy requiring relatively strong and oftenlarge support structures to keep the various lenses of a camera assemblyin alignment. The heavy weight of large lenses makes cameras with suchlenses difficult and bulky to transport. Furthermore, cameras with largelenses often need a tripod or other support to be used for extendedperiods of time given that the sheer weight of a camera with a largelens can become tiresome for an individual to hold in a short amount oftime.

In addition to weight and size drawbacks, large lenses also have thedisadvantage of being costly. This is because of, among other things,the difficulty in manufacturing large high quality optics and packagingthem in a manner in which they will maintain proper alignment over aperiod of time which may reflect the many years of use a camera lensesis expected to provide.

A great deal of effort has been directed in the camera industry tosupporting the use of large camera lenses and packaging them in a waythat allows different lenses to be used in an interchangeable manner ona camera body. However, for the vast majority of camera users, thedrawbacks to cameras with large lenses means that camera users tend notto use large lenses with such lenses often being left to professionalsand/or photo enthusiasts willing to incur the expense and trouble ofbuying and using large lenses.

In fact, many camera owners who own cameras with large high qualitylenses often find themselves taking pictures with small pocket sizecameras, often integrated into other devices such as their cell phones,personal digital assistants or the like, simply because they are moreconvenient to carry. For example, cell phone mounted cameras are oftenmore readily available for use when an unexpected photo opportunityarises or in the case of a general family outing where carrying largebulky camera equipment may be uncomfortable or undesirable.

To frame a given scene from a given point, the focal length (hence size)of the lens depends on the size (area) of the image sensor. The smallerthe image sensor, the smaller the focal length and the smaller the lensrequired. With advances in sensor technology, it is now possible to makesmall sensors, e.g., 5×7 mm² sensors, with relatively high pixel count,e.g., 8 megapixels. This has enabled the embedding of relatively highresolution cameras in small devices such as cell phones. The smallsensor size (compared to larger cameras such as changeable lenssingle-lens reflex (SRL) cameras) enables small focal length lenseswhich are much smaller and lighter than larger focal length lensesrequired for cameras with larger sensors.

Cell phone mounted cameras and other pocket sized digital camerassometimes rely on a fixed focal length lens which is also sometimesreferred to as a focus-free lens. With such lenses the focus is set atthe time of manufacture, and remains fixed. Rather than having a methodof determining the correct focusing distance and setting the lens tothat focal point, a small aperture fixed-focus lens relies on a largedepth of field which is sufficient to produce acceptably sharp images.Many cameras, including those found on most cell phones, with focus freelenses also have relatively small apertures which provide a relativelylarge depth of field. There are also some high end cell phones that useauto focus cameras.

For a lens of a digital camera to be useful, it needs to be paired witha device which detects the light passing through the lens and convertsit to pixel (picture element) values. A megapixel (MP or Mpx) is onemillion pixels. The term is often used to indicate the number of pixelsin an image or to express the number of image sensor elements of adigital camera where each sensor element normally corresponds to onepixel. Multi-color pixels normally include one pixel value for each ofthe red, green, and blue pixel components.

In digital cameras, the photosensitive electronics used as the lightsensing device is often either a charge coupled device (CCD) or acomplementary metal oxide semiconductor (CMOS) image sensor, comprisinga large number of single sensor elements, each of which records ameasured intensity level.

In many digital cameras, the sensor array is covered with a patternedcolor filter mosaic having red, green, and blue regions in anarrangement. In such a filter based approach to capturing a color image,each sensor element can record the intensity of a single primary colorof light. The camera then will normally interpolate the colorinformation of neighboring sensor elements, through a process sometimescalled demosaicing, to create the final image. The sensor elements in asensor array using a color filter are often called “pixels”, even thoughthey only record 1 channel (only red, or green, or blue) of the finalcolor image due to the filter used over the sensor element.

While a filter arrangement over a sensor array can be used to allowdifferent sensor elements to capture different colors of light thusallowing a single sensor to capture a color image, the need to carefullyalign the filter area with individual pixel size sensor elementscomplicates the manufacture of sensor arrays as compared to arrays whichdo not require the use of a multi-color filter array.

While small focal length lenses paired with relatively high resolutionsensors have achieved widespread commercial success in cell phones andpocket cameras, they often leave their owners longing for better picturequality, e.g., picture quality that can only be achieved with a largerpixel area and a larger lens opening to collect more light.

Smaller sensors require smaller focal length lenses (hence smallerlenses) to frame the same scene from the same point. Availability ofhigh pixel count small sensors means that a smaller lens can be used.However, there are a few disadvantages to using smaller sensors andlenses. First, the small pixel size limits the dynamic range of thesensor as only a small amount of light can saturate the sensor. Second,small lenses collect less total light which can result in grainypictures. Third, small lenses have small maximum apertures which makeartistic effects like small depth of field for portrait pictures notpossible.

In view of the above discussion, it should be appreciated that there isa need for improved camera methods and/or apparatus which can provideone or more of the benefits commonly associated with large lenses but ina way that allows a camera to be implemented in a compact manner. Thus,it should be appreciated that there is a need for new photographicmethods and apparatus which can provide some combination of the benefitscommonly associated with large lenses, e.g., a relatively large lensarea for capturing light, with at least some of the benefits of smallfocal length lenses, e.g., compact size. Additionally, it would bedesirable if in some but not necessarily all embodiments thedisadvantages, such as limited dynamic range and/or depth of fieldassociated with small focal length lenses, could be avoided and/or suchadvantages reduced without requiring the use of large lenses.

SUMMARY OF THE INVENTION

Various methods and apparatus relate to implementing a camera in acompact manner by using one or more light redirection devices tofacilitate a compact camera design.

Various features are directed to methods and apparatus for obtainingsome or all of the benefits of using relatively large and long lensassemblies without the need for large lens and/or long lens assemblies,through the use of multiple optical chain modules in combination.

Optical chain modules including, in some embodiments, relatively shortfocal length lenses which require relatively little depth within acamera are used in some embodiments. While use of short focal lengthlens can have advantages in terms of small lens width, the methods andapparatus of the present are not limited to the use of such lenses andcan be used with a wide variety of lens types. In addition, whilenumerous embodiments are directed to autofocus embodiments, fixed focusembodiments are also possible and supported.

An optical chain, in various embodiments, includes a first lens and animage sensor. Additional lenses and/or one or more optical filters maybe included between the first lens of an optical chain module and theimage sensor depending on the particular embodiment. In some cases theremay be one or more optical filters before the first lens.

The use of multiple optical chain modules is well suited for use indevices such as cell phones and/or portable camera devices intended tohave a thin form factor, e.g., thin enough to place in a pocket orpurse. By using multiple optical chains and then combining the capturedimages or portions of the captured images to produce a combined image,improved images are produced as compared to the case where a singleoptical chain module of the same size is used.

While in various embodiments separate image sensors are used for each ofthe individual optical chain modules, in some embodiments the imagesensor of an individual optical chain module is a portion of a CCD orother optical sensor dedicated to the individual optical chain modulewith different portions of the same sensor serving as the image sensorsof different optical chain modules.

In various embodiments, images of a scene area are captured by differentoptical chain modules and then subsequently combined either by theprocessor included in the camera device which captured the images or byanother device, e.g., a personal or other computer which processes theimages captured by the multiple optical chains after offloading from thecamera device which captured the images.

The combined image has, in some embodiments a dynamic range that islarger than the dynamic range of an individual image used to generatethe combined image.

In some such embodiments the sensors of multiple optical chains aremounted on a flat printed circuit board or backplane device. The printedcircuit board, e.g. backplane, can be mounted or coupled to horizontalor vertical actuators which can be moved in response to detected cameramotion, e.g., as part of a shake compensation process which will bediscussed further below. In some such embodiments, pairs of lightdiverting devices, e.g., mirrors, are used to direct the light so thatat least a portion of each optical chain extends perpendicular orgenerally perpendicular to the input and/or sensor plane. Suchembodiments allow for relatively long optical paths which take advantageof the width of the camera by using mirrors or other light divertingdevices to alter the path of light passing through an optical chain sothat at least a portion of the light path extends in a directionperpendicular or generally perpendicular to the front of the cameradevice. The use of mirrors or other light diverting devices allows thesensors to be located on a plane at the rear or front of the cameradevice as will now be discussed in detail.

Numerous additional features and embodiments are described in thedetailed description which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an exemplary block diagram of an exemplary apparatus, e.g.,camera device, implemented in accordance with one embodiment of thepresent invention.

FIG. 1B illustrates a frontal view of an apparatus implemented inaccordance with an exemplary embodiment of the present invention whichincorporates multiple optical chain modules in accordance with thepresent invention with lenses which are viewable from the front of thecamera.

FIG. 1C, which is a side view of the exemplary apparatus of FIG. 1B,illustrates further details of the exemplary apparatus.

FIG. 2 illustrates a camera device implemented in accordance with oneembodiment of the present invention.

FIG. 3A shows an exemplary lens configuration which may be used for theset of outer lenses of the camera device shown in FIGS. 1A-1C.

FIG. 3B illustrates an exemplary filter arrangement which is used in thecamera of FIGS. 1A-1C in some embodiments.

FIG. 3C shows an exemplary inner lens configuration which may, and insome embodiments is, used for a set of inner lenses of the camera deviceshown in FIGS. 1A-1C.

FIG. 4 illustrates an exemplary camera device in which the sets of outerlenses, filters, and inner lenses are mounted on corresponding platters.

FIG. 5A illustrates various filter and lens platters that may be used inthe camera device shown in FIG. 4 depending on the particularembodiment.

FIG. 5B illustrates the filter platter arrangement shown in FIG. 5A whenviewed from the side and when viewed from the front.

FIG. 6, which comprises the combination of FIGS. 6A, 6B, and 6C, showsan exemplary combination of lenses and filters used in one exemplaryembodiment in which a single color filter is used in at least some ofthe different optical chain modules.

FIG. 7, which comprises the combination of FIGS. 7A, 7B, and 7C, showsan exemplary combination of lenses and filters used in one exemplaryembodiment in which exposures of different duration are used fordifferent optical chain modules and a single color filter is used in atleast some of the different optical chain modules.

FIG. 8 illustrates an optical chain arrangement used in one panoramiccamera embodiment in which multiple optical chains and different lensangles are used to capture images that are well suited for combininginto a panoramic image.

FIG. 9 illustrates an exemplary method of producing at least one imageof a first scene area by operating a plurality of optical chain modulesin accordance with one embodiment of the present invention.

FIG. 10 illustrates an exemplary method of producing at least one imageof a first scene area with an enhanced sensor dynamic range by operatingtwo or more optical chain modules in accordance with one embodiment ofthe present invention.

FIG. 11 illustrates an exemplary method of producing at least one imageof a first scene area with enhanced sensor dynamic range by operatingtwo or more optical chain modules in accordance with one embodiment ofthe present invention.

FIG. 12 illustrates an exemplary method of producing at least one imageof a first scene area by operating two or more optical chain modulesusing color filters in accordance with one embodiment of the presentinvention.

FIG. 13 illustrates an exemplary assembly of modules, which may, and insome embodiments is, part of an apparatus which implements one or moremethods of the invention, for performing various image and dataprocessing functions in accordance with one or more exemplaryembodiments of the invention.

FIG. 14 illustrates a computer system which can be used for postprocessing of images captured using a camera device.

FIG. 15 illustrates a frontal view of an apparatus implemented inaccordance with one embodiment of the present invention whichincorporates multiple optical chain modules, e.g., one for each of red,green and blue and one for all three colors.

FIG. 16 illustrates a frontal view of the outer lens assembly of anapparatus implemented in accordance with one embodiment of the presentinvention where the apparatus incorporates multiple optical chainmodules and outer lenses configured with little or no gaps between thelenses.

FIG. 17 illustrates a frontal view of the outer lenses of a lensassembly implemented in accordance with one embodiment of the presentinvention where the apparatus incorporates multiple optical chainmodules with lenses configured with little or no gaps between the lensesbut non-uniform spacing between the optical centers of at least some ofthe lenses.

FIG. 18 illustrates a camera device including a plurality of opticalchain modules which includes mirrors or another device for changing theangle of light entering the optical chain module and thereby allowing atleast a portion of the optical chain module to extend in a direction,e.g., a perpendicular direction, which is not a straight front to backdirection with respect to the camera device.

FIG. 19 illustrates another camera device including a plurality ofoptical chain modules which includes mirrors or another device forchanging the angle of light entering the optical chain module andthereby allowing at least a portion of the optical chain module toextend in a direction, e.g., a perpendicular direction, which is not astraight front to back direction with respect to the camera device.

FIG. 20 illustrates an additional exemplary camera device in whichmirrors and/or other light redirecting elements are used to alter thepath of light in the optical chains so that both the input lenses and/oropenings through which light enters the optical chains can be arrangedin a plane, and also so that the optical sensors of the optical chainscan be arranged in a plane, while allowing at least a portion of thelight path through the optical chains to extend in a directionperpendicular to the input and/or output planes.

FIG. 21 illustrates an additional exemplary camera device in whichmirrors and/or other light redirecting elements are used to alter thepath of light in the optical chains so that the input lenses and/oropenings, as well as the light sensors of the different optical chains,can be arranged in one or more planes at the front of the camera.

DETAILED DESCRIPTION

FIG. 1A illustrates an exemplary apparatus 100, sometimes referred tohereinafter as a camera device, implemented in accordance with oneexemplary embodiment of the present invention. The camera device 100, insome embodiments, is a portable device, e.g., a cell phone or tabletincluding a camera assembly. In other embodiments, it is fixed devicesuch as a wall mounted camera.

FIG. 1A illustrates the camera device 100 in block diagram form showingthe connections between various elements of the apparatus 100. Theexemplary camera device 100 includes a display device 102, an inputdevice 106, memory 108, a processor 110, a transceiver interface 114,e.g., a cellular interface, a WIFI interface, or a USB interface, an I/Ointerface 112, and a bus 116 which are mounted in a housing representedby the rectangular box touched by the line leading to reference number100. The input device 106 may be, and in some embodiments is, e.g.,keypad, touch screen, or similar device that may be used for inputtinginformation, data and/or instructions. The display device 102 may be,and in some embodiments is, a touch screen, used to display images,video, information regarding the configuration of the camera device,and/or status of data processing being performed on the camera device.In the case where the display device 102 is a touch screen, the displaydevice 102 serves as an additional input device and/or as an alternativeto the separate input device, e.g., buttons, 106. The I/O interface 112couples the display 102 and input device 106 to the bus 116 andinterfaces between the display 102, input device 106 and the otherelements of the camera which can communicate and interact via the bus116. In addition to being coupled to the I/O interface 112, the bus 116is coupled to the memory 108, processor 110, an optional autofocuscontroller 132, a transceiver interface 114, and a plurality of opticalchain modules 130, e.g., N optical chain modules. In some embodiments Nis an integer greater than 2, e.g., 3, 4, 7 or a larger value dependingon the particular embodiment. Images captured by individual opticalchain modules in the plurality of optical chain modules 130 can bestored in memory 108, e.g., as part of the data/information 120 andprocessed by the processor 110, e.g., to generate one or more compositeimages. Multiple captured images and/or composite images may beprocessed to form video, e.g., a series of images corresponding to aperiod of time. Transceiver interface 114 couples the internalcomponents of the camera device 100 to an external network, e.g., theInternet, and/or one or more other devices e.g., memory or stand alonecomputer. Via interface 114 the camera device 100 can and does outputdata, e.g., captured images, generated composite images, and/orgenerated video. The output may be to a network or to another externaldevice for processing, storage and/or to be shared. The captured imagedata, generated composite images and/or video can be provided as inputdata to another device for further processing and/or sent for storage,e.g., in external memory, an external device or in a network.

The transceiver interface 114 of the camera device 100 may be, and insome instances is, coupled to a computer so that image data may beprocessed on the external computer. In some embodiments the externalcomputer has a higher computational processing capability than thecamera device 100 which allows for more computationally complex imageprocessing of the image data outputted to occur on the externalcomputer. The transceiver interface 114 also allows data, informationand instructions to be supplied to the camera device 100 from one ormore networks and/or other external devices such as a computer or memoryfor storage and/or processing on the camera device 100. For example,background images may be supplied to the camera device to be combined bythe camera processor 110 with one or more images captured by the cameradevice 100. Instructions and/or data updates can be loaded onto thecamera via interface 114 and stored in memory 108.

The camera device 100 may include, and in some embodiments does include,an autofocus controller 132 and/or autofocus drive assembly 134. Theautofocus controller 132 is present in at least some autofocusembodiments but would be omitted in fixed focus embodiments. Theautofocus controller 132 controls adjustment of at least one lensposition in the optical chain modules used to achieve a desired, e.g.,user indicated, focus. In the case where individual drive assemblies areincluded in each optical chain module, the autofocus controller 132 maydrive the autofocus drive of various optical chain modules to focus onthe same target. As will be discussed further below, in some embodimentslenses for multiple optical chain modules are mounted on a singleplatter which may be moved allowing all the lenses on the platter to bemoved by adjusting the position of the lens platter. In some suchembodiments the autofocus drive assembly 134 is included as an elementthat is external to the individual optical chain modules with the driveassembly 134 driving the platter including the lenses for multipleoptical chains under control of the autofocus controller 132. While theoptical chain modules will in many embodiments be focused together tofocus on an object at a particular distance from the camera device 100,it is possible for different optical chain modules to be focused todifferent distances and in some embodiments different focus points areintentionally used for different optical chains to increase the postprocessing options which are available.

The processor 110 controls operation of the camera device 100 to controlthe elements of the camera device 100 to implement the steps of themethods described herein. The processor may be a dedicated processorthat is preconfigured to implement the methods. However, in manyembodiments the processor 110 operates under direction of softwaremodules and/or routines stored in the memory 108 which includeinstructions that, when executed, cause the processor to control thecamera device 100 to implement one, more or all of the methods describedherein. Memory 108 includes an assembly of modules 118 wherein one ormore modules include one or more software routines, e.g., machineexecutable instructions, for implementing the image capture and/or imagedata processing methods of the present invention. Individual stepsand/or lines of code in the modules of 118 when executed by theprocessor 110 control the processor 110 to perform steps of the methodof the invention. When executed by processor 110, the data processingmodules 118 cause at least some data to be processed by the processor110 in accordance with the method of the present invention. Theresulting data and information (e.g., captured images of a scene,combined images of a scene, etc.) are stored in data memory 120 forfuture use, additional processing, and/or output, e.g., to displaydevice 102 for display or to another device for transmission, processingand/or display. The memory 108 includes different types of memory forexample, Random Access Memory (RAM) in which the assembly of modules 118and data/information 120 may be, and in some embodiments are stored forfuture use. Read only Memory (ROM) in which the assembly of modules 118may be stored for power failures. Non-volatile memory such as flashmemory for storage of data, information and instructions may also beused to implement memory 108. Memory cards may be added to the device toprovide additional memory for storing data (e.g., images and video)and/or instructions such as programming. Accordingly, memory 108 may beimplemented using any of a wide variety of non-transitory computer ormachine readable mediums which serve as storage devices.

Having described the general components of the camera device 100 withreference to FIG. 1A, various features relating to the plurality ofoptical chain modules 130 will now be discussed with reference to FIGS.1B and 1C which show the camera device 100 from front and sideperspectives, respectively.

FIG. 1B shows the front of the camera device 100. Rays of light 131shown in FIG. 1C may enter the lenses located in the front of the camerahousing. From the front of camera device 100, the camera device 100appears as a relatively flat device with the outer rectanglerepresenting the camera housing and the square towards the center of thecamera representing the portion of the front camera body in which theplurality of optical chain modules 130 is mounted.

The front of the plurality of optical chain modules 130 is visible inFIG. 1B with the outermost lens of each optical chain module appearingas a circle represented using a solid line. In the FIG. 1B example, theplurality of optical chain modules 130 include seven optical chainmodules OCM1 through OCM 7 which include lenses represented by the solidcircles shown in FIG. 1B. The lenses of the optical chain modules arearranged to form a pattern which is generally circular in the FIG. 1Bexample when viewed as a unit from the front. While a circulararrangement is preferred in some embodiments, non-circular arrangementsare used and preferred in other embodiments. In some embodiments whilethe overall pattern is generally or roughly circular, differentdistances to the center of the general circle and/or different distancesfrom one lens to another is intentionally used to facilitate generationof a depth map and block processing of images which may include periodicstructures such as repeating patterns without the need to identify edgesof the repeating pattern. Such repeating patterns may be found in agrill or a screen.

Note that the individual outer lenses, in combination, occupy an areathat might otherwise have been occupied by a single large lens. Thus,the overall total light capture area corresponding to the multiplelenses of the plurality of chain modules OCM 1 to OCM 7, also sometimesreferred to as optical camera modules, approximates that of a lenshaving a much larger opening but without requiring a single lens havingthe thickness which would normally be necessitated by the curvature of asingle lens occupying the area which the lenses shown in FIG. 1B occupy.

While gaps are shown between the lens openings of the optical chainmodules OCM 1 to OCM 7, it should be appreciated that the lenses may bemade, and in some embodiments are, made so that they closely fittogether minimizing gaps between the lenses represented by the circlesformed by solid lines. While seven optical chain modules are shown inFIG. 1B, it should be appreciated that other numbers of optical chainmodules are possible.

As will be discussed below, the use of seven optical chain modulesprovides a wide degree of flexibility in terms of the types of filtercombinations and exposure times that can be used for different colorswhile still providing an optical camera module that can be used toprovide an image for purposes of user preview of the image area andselection of a desired focal distance, e.g., by selecting an object inthe preview image which is to be the object where the camera modules areto be focused.

For example, in some embodiments, such as the FIG. 6 embodiment, atleast some of the different optical chain modules include filterscorresponding to a single color thereby allowing capture of a singlecolor at the full resolution of the image sensor, e.g., the sensor doesnot include a Bayer filter. In one embodiment two optical chain modulesare dedicated to capturing red light, two optical chain modules arededicated to capturing green light and two optical chain modules arededicated to capturing blue light. The center optical chain module mayinclude a RGB filter or opening which passes all colors with differentportions of the sensor of the center optical chain module being coveredby different color filters, e.g., a Bayer pattern with the optical chainmodule being used to capture all three colors making it easy to generatecolor preview images without having to process the output of multipleoptical chain modules to generate a preview image.

The use of multiple optical chains such as shown in the FIG. 1A-1Cembodiment has several advantages over the use of a single opticalchain.

Using multiple optical chains allows for noise averaging. For example,given the small sensor size there is a random probability that oneoptical chain may detect a different number, e.g., one or more, photonsthan another optical chain. This may represent noise as opposed toactual human perceivable variations in the image being sensed. Byaveraging the sensed pixel values corresponding to a portion of animage, sensed by different optical chains, the random noise may beaveraged resulting in a more accurate and pleasing representation of animage or scene than if the output of a single optical chain was used.

As should be appreciated, different wavelengths of light will be bent bydifferent amounts by the same lens. This is because the refractive indexof glass (or plastic) which the lens is made of changes with wavelength.Dedication of individual optical chains to a particular color allows forthe lenses for those optical chains to be designed taking intoconsideration the refractive index of the specific range of wavelengthfor that color of light. This can reduce chromatic aberration andsimplify lens design. Having multiple optical chains per color also hasthe advantage of allowing for different exposure times for differentoptical chains corresponding to a different color. Thus, as will bediscussed further below, a greater dynamic range in terms of lightintensity can be covered by having different optical chains usedifferent exposure times and then combining the result to form thecomposite image, e.g., by weighting the pixel values output by thesensors of different optical chains as a function of exposure time whencombing the sensed pixel values to generate a composite pixel value foruse in a composite image. Given the small size of the optical sensors(pixels) the dynamic range, in terms of light sensitivity, is limitedwith the sensors becoming easily saturated under bright conditions. Byusing multiple optical chains corresponding to different exposure timesthe dark areas can be sensed by the sensor corresponding to the longerexposure time while the light areas of a scene can be sensed by theoptical chain with the shorter exposure time without getting saturated.Pixel sensors of the optical chains that become saturated as indicatedby a pixel value indicative of sensor saturation can be ignored, and thepixel value from the other, e.g., less exposed, optical chain can beused without contribution from the saturated pixel sensor of the otheroptical chain. Weighting and combining of non-saturated pixel values asa function of exposure time is used in some embodiments. By combiningthe output of sensors with different exposure times a greater dynamicrange can be covered than would be possible using a single sensor andexposure time.

FIG. 1C is a cross section perspective of the camera device 100 shown inFIGS. 1A and 1B. Dashed line 101 in FIG. 1B shows the location withinthe camera device to which the cross section of FIG. 1C corresponds.From the side cross section, the components of the first, seventh andfourth optical chains are visible.

As illustrated in FIG. 1C despite including multiple optical chains thecamera device 100 can be implemented as a relatively thin device, e.g.,a device less than 2, 3 or 4 centimeters in thickness in at least someembodiments. Thicker devices are also possible, for example devices withtelephoto lenses and are within the scope of the invention, but thethinner versions are particularly well suited for cell phones and/ortablet implementations.

As illustrated in the FIG. 1C diagram, the display device 102 may beplaced behind the plurality of optical chain modules 130 with theprocessor 110, memory and other components being positioned, at least insome embodiments, above or below the display and/or optical chainmodules 130. As will be discussed below, and as shown in FIG. 1C, eachof the optical chains OCM 1, OCM 7, OCM 4 may, and in some embodimentsdo, include an outer lens L1, an optional filter F, and a secondoptional lens L2 which proceed a sensor S which captures and measuresthe intensity of light which passes through the lens L1, filter F andsecond lens L2 to reach the sensor S. The filter may be a color filteror one of a variety of other types of light filters.

In FIG. 1C, each optical chain module includes an auto focus drive (AFD)also sometimes referred to as an auto focus device which can alter theposition of the second lens L2, e.g., move it forward or back, as partof a focus operation. An exposure control device (ECD) which controlsthe light exposure time of the sensor to which the ECD corresponds, isalso included in each of the OCMs shown in the FIG. 1C embodiment. TheAFD of each optical chain module operates under the control of theautofocus controller 132 which is responsive to user input whichidentifies the focus distance, e.g., by the user highlighting an objectin a preview image to which the focus is to be set. The autofocuscontroller while shown as a separate element of the device 100 can beimplemented as a module stored in memory and executed by processor 110.

Note that while supporting a relatively large light capture area andoffering a large amount of flexibility in terms of color filtering andexposure time, the camera device 100 shown in FIG. 1C is relatively thinwith a thickness that is much less, e.g., ⅕th, 1/10th, 1/20th or evenless than the overall side to side length or even top to bottom lengthof the camera device visible in FIG. 1B.

FIG. 2 illustrates a camera device 200 implemented in accordance withthe invention. The FIG. 2 device includes many or all of the sameelements shown in the device 100 of FIGS. 1A-1C. In the FIG. 2embodiment the optical chain modules are shown as independent assemblieswith the autofocus drive of each module being a separate AFD element.

In FIG. 2, the structural relationship between the various lenses andfilters which precede the sensor in each optical chain module can beseen more clearly. While three elements, e.g. two lenses (see columns201 and 203 corresponding to L1 and L2, respectively) and the filter(corresponding to column 202) are shown in FIG. 2 before each sensor, itshould be appreciated that a much larger combination of lenses and/orfilters may precede the sensor of one or more optical chain modules withanywhere from 2-10 elements being common and an even larger number ofelements being used in some embodiments, e.g., high end embodimentsand/or embodiments supporting a large number of filter and/or lensoptions.

In some but not all embodiments, optical chain modules are mounted inthe camera device to extend from the front of the camera device towardsthe back, e.g., with multiple optical chain modules being arranged inparallel. Filters and/or lenses corresponding to different optical chainmodules may, and in some embodiments are, arranged in planes extendingperpendicular to the front to back direction of the camera device fromthe bottom of the camera device towards the top of the camera device.While such a mounting arrangement is used in some embodiments, otherarrangements where the optical chain modules are arranged at differentangles to one another and/or the camera body are possible.

Note that the lenses/filters are arranged in planes or columns in thevertical dimension of the camera device to which reference numbers 201,202, 203 correspond. The fact that the lenses/filters are aligned alongvertical planes allows for a manufacturing and structural simplificationthat is used in some embodiments. That is, in some embodiments, thelenses and/or filters corresponding to a plane 201, 202, 203 are formedor mounted on a platter or plate. The term platter will be used fordiscussion purposes but is not intended to be limiting. The platter maytake the form of a disc but non-round platters are also contemplated andare well suited for some embodiments. In the case of plastic lenses, thelenses and platter may be molded out of the same material in a singlemolding operation greatly reducing costs as compared to the need tomanufacture and mount separate lenses. As will be discussed further,platter based embodiments allow for relatively simple synchronized focusoperations in that a platter may be moved front or back to focusmultiple OCMs at the same time. In addition, as will be explained,platters may be moved or rotated, e.g., along a central or non-centralaxis, to change lenses and or filters corresponding to multiple opticalchain modules in a single operation. A single platter may include acombination of lenses and/or filters allowing, e.g., a lens to bereplaced with a filter, a filter to be replaced with a lens, a filter orlens to be replaced with an unobstructed opening. As should beappreciated the platter based approach to lens, filter and/or holesallows for a wide range of possible combinations and changes to be madeby simple movement of one or more platters. It should also beappreciated that multiple elements may be combined and mounted togetheron a platter. For example, multiple lenses, filters and/or lens-filtercombinations can be assembled and mounted to a platter, e.g., oneassembly per optical chain module. The assemblies mounted on the platterfor different optical chains may be moved together, e.g., by rotatingthe platter, moving the platter horizontally or vertically or by movingthe platter using some combination of one or more such movements.

While platters have been described as being moved to change elements inan optical chain, they can, and in some embodiments are, moved for imagestabilization purposes. For example, a platter having one or more lensesmounted thereon can be moved as part of an image stabilizationoperation, e.g., to compensate for camera motion.

While mounting of lenses and filters on platters has been discussed, itshould also be appreciated that the sensors of multiple optical chainscan be mounted on a platter. For example, sensors without color filtersmay be replaced with sensors with color filters, e.g., Bayer patternfilters. In such an embodiment sensors can be swapped or changed whileleaving one or more components of one or more optical chains in place.

Note from a review of FIG. 2 that in some embodiments, e.g., largerfocal length telephoto applications, the elements, e.g., filters/lensescloser to the sensor of the optical chain module, are smaller in sizethan the outer most lenses shown in column 201. As a result of theshrinking size of the lenses/filters, space becomes available betweenthe lenses/filters within the corresponding platter.

FIGS. 3A through 3C provide perspective views of the different planes201, 202, 203 shown in FIG. 2. As shown in FIG. 3A, the outer lenses L1occupy much of the outer circular area corresponding to the front of thecamera modules as previously shown in FIG. 1B. However, as shown in FIG.3B the filters corresponding to plane 202 occupy less space than thelenses shown in FIG. 3A while the inner lenses L2 shown in FIG. 3Coccupy even less space.

The decreasing size of the inner components allow multiple lenses and/orfilters to be incorporated into a platter corresponding to one or moreof the inner planes. Consider for example that an alternative filter F′or hole could be mounted/drilled below or next two each filter F of aplatter corresponding to plan 202 and that by shifting the position orplatter vertically, horizontally or a combination of horizontally andvertically, the filter F can be easily and simply replaced with anotherfilter or hole. Similarly the lenses L2 may be replaced by alternativelenses L2′ by shifting a platter of lenses corresponding to plane 203.In some embodiments, the platter may also be rotated to support changes.The rotation may be an off center rotation and/or may be performed incombination with one or more other platter position changes.

A camera device 60 which includes platters of lenses and/or filters isshown in FIG. 4. Element 61 represents a platter of outer lenses L1 with3 of the lenses being shown as in the FIG. 1C example. Additional lensesmay be, and often are, included on the platter 61 in addition to theones shown. For example, in a seven optical chain module embodiment suchas shown in FIG. 1, platter 61 would include seven outer lenses. Notethat the thickness of the platter 61 need not exceed the maximumthicknesses of the lenses and from a side perspective is much thinnerthan if a single lens having a similar curvature to that of theindividual lenses L1, but with the single lens being larger, occupiedthe same area as all the 7 lenses on the platter 61. Platter 62 includesthe filters F while platter 63 includes the inner lenses L2. As can beappreciated the camera device 60 is the same as or similar to the cameradevice of FIG. 1C and FIG. 2 but with the lenses and filters beingmounted on platters which may be moved between the front and back of thecamera to support autofocus or horizontally and/or vertically to supportlens/filter changes.

Auto focus drive 66 is used to move platter 63 forward or backward aspart of a focus operation, e.g., under control of the autofocuscontroller 132 which may be, and often is, included in the camera device60. A filter shift drive (FSD) 65 is included in embodiments whereshifting of the platter 62 is supported as part of a filter changeoperation. The FSD 65 is responsive to the processor 110 which operatesin response to user selection of a particular mode of operation and/oran automatically selected mode of operation and can move the platter 62vertically, horizontally or in some combination of vertical andhorizontal motion to implement a filter change operation. The FSD may beimplemented with a motor and mechanical linkage to the platter 62. Insome embodiments, the platter 62 may also be rotated to support changes.The rotation may be an off center rotation and/or may be performed incombination with one or more other platter position changes.

A lens shift drive (LSD) 67 is included in embodiments where shifting ofthe platter 63 is supported as part of a filter change operation. TheLSD 67 is responsive to the processor 110 which operates in response touser selection of a particular mode of operation and/or an automaticallyselected mode of operation and can move the platter 63 vertically,horizontally or in some combination of vertical and horizontal motion toimplement a lens change operation. The LSD 67 may be implemented with amotor and mechanical linkage to the platter 63. In some embodiments, theplatter 63 may also be rotated to support changes. The rotation may bean off center rotation and/or may be performed in combination with oneor more other platter position changes.

FIG. 5A illustrates various exemplary platters that can, and in someembodiments are, used as the filter platter and/or inner lens platter inthe camera device 60 of FIG. 4. In the FIG. 5A example N is three (3)but other values of N are possible depending on the embodiment. FIG. 5Bshows the exemplary lens platter 62′ of FIG. 5A when viewed from theside and from the front.

Platter 62 represents a platter with a single set of filters F1,1corresponding to OCM1, F1,2 corresponding to OCM 2 and F1,3corresponding to OCM 3.

Platter 62′ represents an alternative platter that can, and in someembodiments is, used in place of platter 62. NF is use to represent ahole or No Filter (NF) area of the platter 62′. As should be appreciatedby simply shifting platter 62′ vertically the filters F1 can be replacedby holes thereby removing the color or other types of filters previouslyincluded in the optical chain modules.

Platter 62″ of FIG. 5A represents a platter which includes alternativefilters F1′ which can be switched for the filters F1 by moving thepalter 62″ vertically. Thus platter 62″ is used to show how filters canbe switched for other filters by simple movement of a platter whileplatter 62′ shows how filters can be removed from the optical pathsincluded in a plurality of optical chain modules by shifting of theplatter on which a set of filters are mounted.

Lens platter 63 shows a platter of inner lenses L2 corresponding tofirst, second and third optical camera modules. Lens platter 63′ is analternative platter which shows how alternative lenses L2′ can beincluded on a lens platter and easily swapped for the lenses L2 bysimple movement of the platter 63′ vertically or horizontally. Lensplatter 63″ is used to show that a lens platter may include holes as analternative to alternative lenses. Any of lens platters 63, 63′ or 63″could be used in the camera device 60 shown in FIG. 4. While two lenssets are included in platter 63′, multiple lens and/or holecombinations, e.g., 2, 3 or more, may be included in a single platter.Similarly a large number of alternative filter, hole alternatives may besupported in a single filter platter. A platter can also havecombinations of lenses, filters and holes and filters could be swappedfor lenses or holes.

As should be appreciated given the larger number of lens/filtercombinations that can be supported through the use of platters, a singlecamera device including a number of optical chain modules may support alarge number of alternative modes of operation.

It should be appreciated that the exposure control of various opticalchain modules may be varied along with the filters and/or lenses used atany given point in time allowing for a wide degree of flexibility andcontrol over the images captured at any given point in time.

FIGS. 6A, 6B and 6C corresponding to one particular filter lenscombination used in some embodiments.

FIG. 6A shows the use of 7 optical chain modules at plane 201 (the outerlens plane corresponding to lenses L1) as viewed from the front of thecamera device. FIG. 6C shows the inner lens plane 203. The configurationshown in FIGS. 6A and 6C is the same or similar to that previouslydiscussed with reference to the FIG. 3 embodiment. FIG. 6B shows aparticular color filter arrangement used in some embodiments. The filterarrangement shown in FIG. 6B may be used in the set of optical chainmodules 130 before the sensors, e.g., between the set of L1 and L2lenses. However, this position is not required for some embodiments andthe user of inner lenses L2 is also not required for some embodiments.

The filter configuration 602 includes single color filters in each of aplurality of optical chain modules, e.g., the six outer optical chainmodules (OCM1 to OCM6). Multiple optical chain modules are dedicated toeach of the three colors, red (R), green (G) and blue (B). The opticalchain modules (OCM1, OCM4) with the red filter (RF) pass and sense redlight. The optical chain modules (OCM 2, OCM 5) with the green filter(GF) pass and sense green light. The optical chain modules (OCM 3, OCM6) with the blue filter (BF) pass and sense blue light.

By using optical chain modules dedicated to a single color, the opticalchains can be optimized for the spectral range corresponding to theparticular color to which the chain corresponds. In addition postcapture color compensation can be simplified since each of the six outeroptical modules capture a single known color. In addition, noise can beaveraged between the sensor corresponding to the same color and/ordifferent exposure times can be used for the different OCMscorresponding to an individual color extending the dynamic range of thesensors to cover a range wider than could be captured by a singlesensor. In addition different exposure times may be used for differentcolors to take into consideration particular color biased lightingconditions and/or facilitate the implementation of particular coloreffects that may be desired. Notably the individual colors are capturedat a pixel result in a resolution equal to that of the sensor as opposedto the case where different portions of a single sensor are used tocapture different colors, e.g., with each color R, G, B being capturedat a resolution ⅓ that of the pixel resolution of the image sensor beingused in an optical chain module.

While in some embodiments a composite image is generated and displayedas a preview image, in some embodiments to reduce processing time and/orthe time required to display a composite image which may be delayed bythe time required to combine multiple images, an image captured by asingle sensor is displayed as the preview image on the display of thecamera device. The multi-colored filter incorporated into the sensor,e.g., Bayer filter, of OCM 7 allows a color image to be captured by asingle lens and used as the preview image. While the image may be oflower quality than that which can be generated by creating a compositeof the multiple OCMs given the small display size the difference inimage quality between the preview image generated from OCM 7 and that ofa composite image may not be sufficient to justify the processing,power, and/or time required to generate a composite image for previewpurpose. Accordingly, the FIG. 6B filter arrangement provides a greatdeal of flexibility while being able to support a wide variety ofexposure and other image capture related features.

The ability to use different exposure times with different optical chainmodules is illustrated further with regard to a camera embodiment whichwill now be discussed with regard to FIGS. 7A, 7B and 7C. The lensconfigurations of FIGS. 7A and 7C are similar to that shown in FIGS. 6Aand 6C. The filter arrangement shown in FIG. 7B is also the same orsimilar to that shown in FIG. 6B but in the FIG. 7B example exposuretime is also included. While the exposure is controlled by use of theexposure control device in some embodiments the concept can beunderstood from FIG. 7B. In FIG. 7B SE is used to indicate shortexposure, LE is used to indicate long exposure, and ME is used toindicate medium exposure. The preview image is generated using themedium exposure optical chain module while the two different opticalchain modules corresponding to a given color use different exposures. Inthis way the short exposure time can be used to reliably captureinformation corresponding to light (e.g., bright) portions of an imagewhile the long exposure optical chain module can be used to captureinformation corresponding to the darker portions of an image. Asdiscussed above, the sensed pixel values from the two optical chains canbe processed to exclude values generated by saturated sensors and tocombine pixel values corresponding to the same image area in a mannerweighted according to the exposure duration for pixel value within theacceptable operating range of the optical chain module's sensors.

While different durations can and often are achieved by controllingsensor exposure times, different filters in different optical chainmodules may, and are, used to achieve different light exposures in someembodiments.

FIG. 8, illustrates an optical chain arrangement used in one panoramiccamera device 8000 in which multiple optical chains and different lensangles are used to capture images that are well suited for combininginto a panoramic image. A1 represents a first non-zero angle, Srepresents a straight or 0 degree angle, and A2 represents a secondnon-zero angle. In one embodiment A1 causes the corresponding camerachain module to capture images to the right of the camera, S causes thecorresponding camera chain module to capture images straight ahead ofthe camera, and A2 causes the corresponding camera chain module tocapture images to the left the camera, from the perspective of the userbehind the camera. In addition to captured images left and right of thecamera it should be appreciated that the optical chain modules capturesome image portion which is also captured by the adjacent optical chainmodule. Thus, the OCMs in columns 803, 805 and 807 capture differentscenes which, while overlapping, can be stitched together to provide anultra wide angle panoramic image. The OCMs in each of rows 811, 813,815, 817, 819, 821, 823 capture different versions of the same scene.

The panoramic camera device 8000 includes multiple optical chain modulescorresponding to each of the left, right and center views. Twenty oneoptical chain modules (seven sets of three) are shown allowing for twooptical chain modules per color (R, G, B) plus a seventh multi-color (R,G, B) optical chain module which can be used to support a preview modeof operation. The multi-color optical chain module may include a sensorwith a multicolor filter, e.g., a Bayer pattern filter, allowing thesingle sensor to capture the multiple colors using different portions ofthe sensor. While the panoramic configuration shown in FIG. 8 isdifferent from that of the non-panoramic camera embodiments previouslydiscussed the exposure control and separate color capture benefitsremain the same as those discussed with regard to the other embodiments.

While FIG. 8 illustrates a particular panoramic embodiment, it should beappreciated that embodiments such as those shown in FIGS. 3 and 4 can,and in sometimes are, used to support taking of panoramic pictures. Inone such embodiment a prism or angled lens is inserted into one or moreoptical chain modules, e.g., by rotation, vertical movement, horizontalmovement and/or a combination of vertical and horizontal movement of aplatter upon which the prism or lens is mounted. The prisms or changesin lens angles change the scene area perceived by one or more opticalchain modules allowing the different optical chain modules to capturedifferent views of a scene which can, and in some embodiments are, usedto generate a panoramic image, e.g., picture. Thus, camera modules usedto capture images corresponding to the same scene which are thencombined to generate a combined image can also be used at a differenttime to capture images corresponding to different views and/or sceneswhich can then be subsequently combined to form a panoramic image, e.g.,photograph.

Accordingly, it should be appreciated that ultra wide angle panoramicimages can be generated using multiple optical chain modules of the typepreviously discussed thereby providing panoramic cameras many of thebenefits of large lens without the need for the camera depth, weight andother disadvantages associated with large lenses.

It should be appreciated that because camera chain modules are separatedfrom one another the multi-optical chain module embodiments of thepresent invention are well suited for stereoscopic image generation andfor generating image depth maps. Accordingly the camera devices of thepresent invention support a wide range of applications and modes ofoperation and provide significant amounts of image data which can beused to support a wide range of post capture image processingoperations.

Having described apparatus and various embodiments, various methodswhich are supported and used in some embodiments will now be discussedwith regard to various flow charts that are included in the presentapplication.

Method 300 of FIG. 9 illustrates one exemplary method of producing atleast one image of a first scene area in accordance with the presentinvention. The processing steps of the method 300 of FIG. 9 will now beexplained in view of the camera device 100 of FIG. 1A.

The method 300 of FIG. 9 starts at start step 302 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 302 to step 304. In step 304, user input isreceived to control the capture of at least one image of the first scenearea. The user input is received via input device 106 which may be, andin some embodiments is, a button or touch sensitive screen. In optionalsub-step 306, the user input may, and in some embodiments does, indicatea portion of the first scene area that is to be focused, e.g., in animage to be captured or a combined image to be generated from two ormore captured images. From step 304 processing proceeds to step 308.

In step 308, a plurality of three or more optical chain modules (OCMs),e.g., optical chain modules 130 of FIG. 1A, are operated in parallel tocapture images of the first scene area, said images including at least afirst image of said first scene area, a second image of said first scenearea, and a third image of said first scene area. In some embodimentseach one of the first, second and third optical chain modules captures acorresponding one of the first, second and third image respectively. Insome embodiments, operating a plurality of three or more optical chainmodules in parallel to capture images of the first scene area, saidimages including at least a first image of said first scene area, asecond image of said first scene area, and a third image of said firstscene area includes sub-processing steps 310, 312, and 314.

In sub-step 310 a first optical chain module is operated to capture afirst image 316 of the first scene area. In most, but not all,embodiments, on capture of the first image 316, the image data and otherdata such as camera device configuration information associated with thefirst image is stored in the data/information 120 portion of memory 108for later processing, output or display. In parallel with the processingof sub-step 310 processing of sub-steps 312 and 314 also occur. Insub-step 312 a second optical chain module is operated to capture asecond image 318 of the first scene area. In most, but not all,embodiments on capture of the second image 318, the image data and otherdata such as camera device configuration information associated with thesecond image is stored in the data/information 120 portion of memory 108for later processing, output or display. In sub-step 314 a third opticalchain module is operated to capture a third image 320 of the first scenearea. In most, but not all, embodiments on capture of the third image320, the image data and other data such as camera device configurationinformation associated with the third image is stored in thedata/information 120 portion of memory 108 for later processing, outputor display. Processing then proceeds from step 308 to step 322.

In some embodiments, each optical chain module of the plurality ofoptical chain modules includes a lens and the lenses of the plurality ofthe optical chain modules are arranged along a circle. For example, whenthere are three optical chain modules, i.e., a first optical chainmodule, a second optical chain module, and a third optical chain module,the first optical chain module includes a first lens, the second opticalchain module includes a second lens, and the third optical chain moduleincludes a third lens. The first, second and third lenses are arrangeduniformly along a circle, e.g. on the vertices of an equilateraltriangle. In some embodiments the camera device 100 includes a fourthoptical chain module including a fourth lens, said fourth lens beingpositioned in the center of the circle. Each of the first, second, thirdand fourth lens may be, and in some embodiments of the present inventionare, the outer lens of each of their respective optical chain modulesand are all positioned in the same plane. More generally, in someembodiments of the present invention, there are a plurality of N opticalchain modules each including a lens. N−1 lenses of the plurality ofoptical chain modules are arranged along a circle with Nth lens beingpositioned in the center of the circle. FIG. 1B illustrates and exampleof a camera device 100 with seven optical chain modules which include 7outer lenses shown as circles, i.e., OCM1, OCM2, OCM3, OCM4, OCM5, OCM6,and OCM7. The outer lens of optical chain modules OCM 1, OCM2, OCM3,OCM4, OCM5, and OCM6 are arranged along a circle and the outer lens ofoptical chain module OCM7 is positioned in the center of the circle.

In some embodiments of the present invention, the first optical chainmodule includes in addition to the first lens an image sensor referredto as a first image sensor. In some embodiments of the presentinvention, the second optical chain module includes an image sensorreferred to as a second image sensor. In some embodiments of the presentinvention, the third optical chain includes an image sensor referred toas a third image sensor. In some embodiments of the present inventionthe plurality of lenses of the plurality of optical chain modules aremounted in a cell phone housing with the plurality of lenses oriented inthe same direction and in the same plane of the housing. For example inthe case of three optical chain modules, in some embodiments of thepresent invention, the first, second and third lenses of the first,second, and third optical chain modules respectively are mounted in acell phone housing and are oriented in the same direction and in thesame plane of the housing.

In step 322, said first, second, and third images are processed byprocessor 110 to generate a first combined image 326 of said first scenearea. In some embodiments, including those embodiments of the presentinvention in which user input is received indicating a portion of thefirst scene area to be focused in the combined image, step 322 may, andin some embodiments does, include sub-step 324 wherein pixel positionson at least one of said first, second, and third images is shifted priorto generating said first combined image to align the portion of thefirst scene to be focused. Processing then proceeds to step 328 wherethe generated combined image is stored in data/information 120 of memory108, e.g., for potential later display, output from the camera device,and/or additional processing and/or displayed on display 102 of cameradevice 100.

In some embodiments, processing step 322 and/or sub-step 324 areperformed on an external device such as a computer. In such cases, thefirst, second and third images are outputted from the camera device 100via transceiver 114 to the external computer for processing to generatethe first combined image 326. The first combined image may then bestored in memory associated with the external device and/or displayed ona display associated with the external computer. In some embodiments ofthe present invention, the first combined image of the first scene areaincludes the same or fewer pixel values than either of said first,second or third images.

From step 328 processing proceeds to step 304 where processing continuesand the method is repeated.

In some embodiments of the present invention, the size of the diameterof the first, second and third lens of the first, second, and thirdoptical chain modules respectively are the same and the sensors of thefirst, second and third optical chain modules have the same number ofpixels. In other embodiments of the present invention, one or moreoptical chain modules may, and in some embodiments do, have lenses withdifferent diameter sizes and/or sensors with different numbers ofpixels. In some embodiments of the present invention, the first, secondand third lenses of the first, second and third optical chain modulesrespectively, are less than 2 cm in diameter and each of the first,second and third image sensors of the first, second and third opticalchain modules support at least 8 Mpixels. In some embodiments of thepresent invention, the first and second lenses are each less than 2 cmin diameter and each of the first and second image sensors support atleast 5 Mpixels. However in many embodiments the image sensors support 8Mpixels or even more and in some embodiments the lenses are larger than2 cm. Various combinations of lens and sensors may be used with avariety of lens sizes being used for different optical chains in someembodiments. In addition different optical chains may use lenses withdifferent shapes, e.g., while the lens may be a spherical lens theperimeter of the lens may be cut into one of a variety of shapes. In oneembodiment, lenses of different optical chain modules are shaped andarranged to minimize gaps between lenses. Such an approach can have theadvantage of resulting in a smoother blur with regard to portions ofcaptured images which are out of focus when combining images captured bydifferent optical chain modules and result in an overall image whichmore closely approximates what might be expected had a single large lensbeen used to capture the scene shown in the combined image.

In accordance with some aspects of the present invention, the diametersize and arrangement of the lenses of the plurality of optical modulesmay and do vary. Similarly the number of pixels supported by the sensorsof each of the plurality of optical modules may also vary for exampledepending on the desired resolution of the optical chain module.

Method 400 of FIG. 10 illustrates an embodiment of a method of producingat least one image of a first scene area in accordance with the presentinvention. The method 400 achieves enhanced sensor dynamic range bycombining images captured through the operation of two or more opticalchain modules using different exposure times. The processing steps ofthe method 400 of FIG. 10 will now be explained in view of the cameradevice 100 of FIG. 1A. For ease of explanation of the method 400, itwill be assumed that the plurality of optical chain module 130 of cameradevice 100 of FIG. 1A includes two optical chain modules and in someembodiments an optional third optical chain module which will bereferred to as a first, second and third optical chain modulerespectively.

The method 400 of FIG. 10 starts at start step 402 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 402 to step 404. In step 404, one of aplurality of optical chain modules of the camera device is operated tocapture an image which will be referred to herein as a fourth image ofthe first scene. For example, one of said first, second or optionalthird optical chain modules may be, and in some embodiments is, operatedto capture the fourth image. This fourth image is captured prior tocapturing the first, second or third images which will be discussed inconnection with step 410 below.

Processing then proceeds to step 406 where the fourth image is displayedon the display 102 of the camera device 100. By displaying the fourthimage on the display of the camera device 100 a user can aim the cameradevice and target the first scene area for which the user wants tocapture an image. In some embodiments, the fourth image is also storedin data/information 120 of memory 108. Processing then proceeds fromstep 406 to step 408.

In step 408, user input is received to control the capture of an imageof the first scene area. The user input is received via input device 106which may be, and in some embodiments is, a button or touch sensitivescreen. For example, the user may touch a portion of the touch sensitivescreen on which the fourth image is shown to focus the camera on aportion of the scene for which an image is to be captured. From step 408processing proceeds to step 410 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 410 includes sub-steps 412, 414, and optional sub-step 416. Insub-step 412, a first optical chain module is operated to capture afirst image 418 of the first scene area using a first exposure time. Insub-step 414, a second optical chain module is operated to capture asecond image 420 of the first scene area using a second exposure time,at least said first and said second exposure times being of differentduration but overlapping in time. In some embodiments, an optionalsub-step 416 is performed wherein a third optical chain module isoperated to capture a third image 422 of the first scene area using athird exposure time. In some embodiments, the third exposure time isdifferent than the first and second exposure times. Additional opticalchain modules may be, and in some embodiments are, used to captureadditional images of the first scene area with the additional opticalchain modules using the same or different exposure times as the first,second or third exposure times so as to obtain additional image data forthe first scene area. Sub-steps 412, 414, and optional sub-step 416 areperformed in parallel so that multiple images of the first scene arecaptured in parallel with different exposure times. The first, secondand optional third captured images may be, and in some embodiments are,stored in data/information 120 of memory section 108 to be available forlater use such as for example in later steps of the method forgenerating a combined image of the first scene area, or for display oroutputting of images.

In some embodiments, in step 404 the operation of one of the first,second and third optical chain modules to capture the fourth image ofthe first scene area uses a fourth exposure time different from saidfirst, second and third exposure times. Once again step 404 occurs priorto the step 410 as the fourth image is displayed on the display 102 sothe user can utilize the displayed image to target the scene area to becaptured by the first, second and optional third images.

Operation of the method proceeds from step 410 to step 424. In step 424the captured images, that is the first and second images, are processedto generate a first combined image of the first scene area 430. In thoseembodiments in which the optional third image was captured optionalsub-step 428 is performed wherein the third image in addition to thefirst and second image is also processed to generate the first combinedimage of the scene area 430.

In some embodiments step 424 is accomplished using sub-step 426 whereinsaid processing of said first and second images and optionally saidthird image to generate a first combined image of the first scene areaincludes combining weighted pixel values of said first image, secondimage, and optional third image. The weighting of the pixel values may,and in some embodiments is a function of exposure times. Thus, at leastin some embodiments, a pixel value of the combined image is generated byweighting and summing a pixel value from each of the first, second andthird images, where the pixel value from the first image is weightedaccording to the first exposure time used to capture the first image,the pixel value from the second image is weighted according the secondexposure time used to capture the second image and the pixel value fromthe third image is weighted according to the third exposure time used tocapture the third image.

Operation proceeds from step 424 to step 432. In step 432, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., touch sensitive display of the camera device 100.

Operation proceeds from step 432 to step 404 where processing continuesand the method is repeated.

In some embodiments of the present invention step 424 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 424 is performed. Step432 is then typically performed by the external device with the combinedimage 430 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 400 may be, and in some embodiments is, implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

Method 500 of FIG. 11 illustrates an embodiment of a method of producingat least one image of a first scene area in accordance with the presentinvention. The method 500 achieves enhanced sensor dynamic range bycombining images captured through the operation of two or more opticalchain modules using different exposure times. The processing steps ofthe method 500 of FIG. 11 will now be explained in view of the cameradevice 100 of FIG. 1A. For ease of explanation of the method 500, itwill be assumed that the plurality of optical chain module 130 of cameradevice 100 of FIG. 1A includes two optical chain modules and in someembodiments an optional third optical chain module which will bereferred to as a first, second and third optical chain modulerespectively. Method 500 is similar to method 400 but implements thecapture of the fourth image and display of the fourth image after thefirst, second and third images have been captured. In this way the userof the device is able to see on the display the first scene area thatwas captured in the first, second and optional third image and whichwill be processed to generate a combined image.

The method 500 of FIG. 11 starts at start step 502 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 502 to step 504. In step 504, user input isreceived to control the capture of the image of the first scene area.The user input is received via input device 106 which may be, and insome embodiments is, a button or touch sensitive screen. From step 504processing proceeds to step 506 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 506 includes sub-steps 510, 512, and optional sub-step 514. Insub-step 510, a first optical chain module is operated to capture afirst image 516 of the first scene area using a first exposure time. Insub-step 512, a second optical chain module is operated to capture asecond image 518 of the first scene area using a second exposure time,at least said first and said second exposure times being of differentduration but overlapping in time. In some embodiments, an optionalsub-step 514 is performed wherein a third optical chain module isoperated to capture a third image 520 of the first scene area using athird exposure time. In some embodiments, the third exposure time isdifferent than the first and second exposure times. Additional opticalchain modules may be, and in some embodiments are, used to captureadditional images of the first scene area with the additional opticalchain modules using the same or different exposure times as the first,second or third exposure times so as to obtain additional image data forthe first scene area and thereby enhancing the effective sensor dynamicrange of the camera device. Sub-steps 510, 512, and optional sub-step514 are performed in parallel so that multiple images of the first sceneare captured in parallel with different exposure times. The first,second and optional third captured images may be, and in someembodiments are, stored in data/information 120 of memory section 108 tobe available for later use such as for example in later steps of themethod for generating a combined image of the first scene area, or fordisplay or outputting of the images. Operation proceeds from step 506 tosteps 522 and 528.

In step 522, one of said first, second and optional third optical chainmodules is operated to capture a fourth image 524 of the first scenearea after capturing one of said first, second and third images. Whilein this particular embodiment the fourth image is captured after thefirst, second and third images, in some embodiments one of the first,second and third images is used as the fourth image. In some embodimentsa fourth exposure time different from said first, second and thirdexposure times is used to capture the fourth image 524. The fourth imagemay be, and in some embodiments is stored in data/information 120 ofmemory 108 for potential later use, output or display. Processingproceeds from step 522 to step 526. In step 526, the fourth image of thefirst scene area is displayed on display 102 of the camera device, e.g.,a touch sensitive screen so that a user of the camera device can see animage of the first scene area that was captured by the first, second andoptional third images. Processing proceeds from step 526 to step 504where processing associated with the method continues as the method isrepeated.

Returning to step 528, in step 528 the first and second images areprocessed to generate a first combined image of the first scene area534. In those embodiments in which the optional third image was capturedoptional sub-step 532 is performed wherein the third image in additionto the first and second images is also processed to generate the firstcombined image of the scene area 534.

In some embodiments step 528 is accomplished using sub-step 530 whereinsaid processing of said first and second images and optionally saidthird image to generate a first combined image of the first scene areaincludes combining weighted pixel values of said first image, secondimage, and optional third image. The weighting of the pixel values may,and in some embodiments is a function of exposure times. Thus, at leastin some embodiments, a pixel value of the combined image is generated byweighting and summing a pixel value from each of the first, second andthird images, where the pixel value from the first image is weightedaccording to the first exposure time used to capture the first image,the pixel value from the second image is weighted according the secondexposure time used to capture the second image and the pixel value fromthe third image is weighted according to the third exposure time used tocapture the third image.

Operation proceeds from step 528 to step 536. In step 536, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., the touch sensitive display of the camera device 100.

Operation proceeds from step 536 to step 504 where processing continuesand the method is repeated.

In some embodiments of the present invention step 528 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 528 is performed. Step536 is then typically performed by the external device with the combinedimage 534 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 500 may be, and in some embodiments, is implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

The use of an external computer to perform some or a part of theprocessing of the first, second and optional third images allows for theuse of computational more complex algorithms as the external computermay be, and in some embodiments does have, a computationally morepowerful processing capability than the camera device 100.

Method 600 of FIG. 12 illustrates an embodiment of a method of producingat least one color image of a first scene area in accordance with thepresent invention. The method 600 uses color filters in connection withcombining two or more images of a first scene area to obtain a colorimage of the first scene area. The processing steps of the method 600 ofFIG. 12 will now be explained in view of the camera device 100 of FIG.1A. For ease of explanation of the method 600, it will be assumed thatthe plurality of optical chain module 130 of camera device 100 of FIG.1A includes two optical chain modules and in some embodiments anoptional third and/or fourth optical chain module which will be referredto as a first, second, third and fourth optical chain modulerespectively.

The method 600 of FIG. 12 starts at start step 602 with the start of thesteps of the method being implemented, e.g., on processor 110. Operationproceeds from start step 602 to step 604. In optional step 604, a fourthoptical chain module of the camera device is operated to capture animage, e.g., a image referred to herein as a fourth image of a firstscene area using a multi-color filter. This fourth image is capturedprior to capturing the first, second or third images which will bediscussed in connection with step 610 below.

Processing then proceeds to optional step 606 where the fourth image isdisplayed on the display 102 of the camera device 100. By displaying thefourth image on the display of the camera device 100 a user can aim thecamera device and target the first scene area for which the user wantsto capture an image. In some embodiments, the fourth image is alsostored in data/information 120 of memory 108. Processing then proceedsfrom step 606 to step 608.

In step 608, user input is received to control the capture of an imageof the first scene area. The user input is received via input device 106which may be, and in some embodiments is, a button or touch sensitivescreen. For example, the user may touch a portion of the touch sensitivescreen on which the fourth image is shown to focus the camera on aportion of the scene for which an image is to be captured. From step 608processing proceeds to step 610 where the plurality of optical chainmodules 130 are operated in parallel to capture images of the firstscene area.

Step 610 includes sub-steps 612, 614, and optional sub-step 616. Insub-step 612, a first optical chain module is operated to capture afirst image 618 of the first scene area using a first color filter. Insub-step 614, a second optical chain module is operated to capture asecond image 620 of the first scene area using a second color filter,said first and said second color filters corresponding to a first colorand a second color respectively. Said first and said second colors beingdifferent colors. In some embodiments, said first and second colorfilters are single color filters which correspond to said first andsecond colors, respectively. In some embodiments, an optional sub-step616 is performed wherein a third optical chain module is operated tocapture a third image 622 of the first scene area using a third colorfilter. In some embodiments, the third color filter corresponds to acolor that is different from said first and second colors. In someembodiments the third color filter is a single color filter whichcorresponds to said third color. Additional optical chain modules maybe, and in some embodiments are, used to capture additional images ofthe first scene area with the additional optical chain modules using thesame or different color filters as the first, second or third colorfilters so as to obtain additional image data for the first scene area.Sub-steps 612, 614, and optional sub-step 616 are performed in parallelso that multiple images of the first scene area are captured in parallelwith different color filters. The first, second and optional thirdcaptured images may be, and in some embodiments are, stored indata/information 120 of memory section 108 to be available for later usesuch as for example in later steps of the method for generating acombined image of the first scene area, or for display or outputting ofimages. In some embodiments of the present invention, the first opticalchain module includes a first lens and a first image sensor and thesecond optical module includes a second lens and a second image sensorand the optional third optical chain module includes a third lens and athird image sensor. In some embodiments, said first and said secondimage sensors are of the same resolution. In some embodiments of thepresent invention, said optional third image sensor of said thirdoptical chain module has the same resolution as the first and secondimage sensors. In some embodiments of the present invention, the fourthoptical chain module includes a fourth lens and a fourth image sensor.In some embodiments of the present invention the fourth image sensor isof the same resolution as the first and second image sensor. In someembodiments of the present invention, the first, second and third lensesof the first, second and third optical chain modules are arranged in acircle, and the fourth lens of the fourth optical chain is arranged inthe center of the circle.

Operation of the method proceeds from step 610 to step 624. In step 624the captured images, that is the first and second images, are processedto generate a first combined image of the first scene area 630. In thoseembodiments in which the optional third image was captured optionalsub-step 628 is performed wherein the third image in addition to thefirst and second images is also processed to generate the first combinedimage of the scene area 630. In some embodiments the fourth image of thefirst scene area is also processed with the first, second and thirdimages to generate the first combined image of the first scene area.

Operation proceeds from step 624 to step 632. In step 632, the generatedfirst combined image of the first scene area is stored indata/information 120 of memory 108 and/or displayed on the display 102,e.g., a touch sensitive display of the camera device 100.

Operation proceeds from step 632 to step 604 where processing continuesand the method is repeated.

In some embodiments of the present invention step 624 is performed on anexternal device such as a computer that is coupled to the camera device100 via the transceiver interface 114. In such embodiments the first,second and optional third images are transmitted to the external devicevia the transceiver interface 114 where the step 624 is performed. Step632 is then typically performed by the external device with the combinedimage 630 being stored in memory associated with the external deviceand/or displayed on a display associated with the external device.

Method 600 may be, and in some embodiments, is implemented on a varietyof devices including for example, a camera or a mobile device such as amobile cellular telephone or a tablet.

In some embodiments of the present invention, each image is presented asit is captured on the display or in the case of a combined image whensaid image has been generated.

In some embodiments of the present invention, each of the capturedimages, e.g., the first, second, third, and fourth images may be, andis, displayed on the display 102 of the camera device 100 as it iscaptured along with one or more combined images that are formed byprocessing and/or combining the first, second, third and/or fourthimages. In some embodiments of the present invention, each of the imagesmay be, is shown, in a separate portion of the display with the size ofthe image being adjusted so that each image displayed is shown in itsentirety. In some embodiments of the present invention, a caption isautomatically placed under each image as it displayed on the screen. Insome embodiments of the present invention, the caption includes thenumber of the image or an indication that it is a combined image, e.g.,image 1, image 2, image 3, image 4, combined image from image 1, 2, 3,and 4. In some embodiments of the present invention, each image ispresented as it is captured on the display or in the case of a combinedimage when said image has been generated. The images may be arranged ina variety of ways on the display 102 after capture and theaforementioned embodiments are only meant to be exemplary in nature.

In some embodiments of the present invention, the image generated bycombining the images captured from two or more of the optical chainmodules is displayed for targeting purposes so that the user may provideinput to control the capture of the image of the scene area and/or theobject in the scene upon which the combined image should be focused.

The FIG. 13 assembly of modules 1700 may, and in some embodiments is,used to process data for example first, second, third and fourth imagesand associated data, and storing and displaying images. In the FIG. 13example, the assembly of modules includes a image processing module1702, a display module 1704, and a storage module 1706. The modulesimplemented one or more of the previously discussed image processingsteps and may include a variety of sub-modules, e.g., an individualcircuit, for performing an individual step of method or methods beingimplemented.

FIG. 14 illustrates a computer system which can be used for postprocessing of images captured using a camera device. The computer system1400 includes a display 1402, Input/Output (I/O) interface 1412,receiver 1404, input device 1406, transceiver interface 1414, processor1410 and memory 1408. The memory is coupled to the processor 1410, I/Ointerface 1412 and transceiver interface 1414 via bus 1416 through whichthe elements of the computer system 1400 can exchange data and cancommunicate with other devices via the I/O interface 1412 and/orinterface 1414 which can couple the system 1400 to a network and/orcamera apparatus. It should be appreciated that via interface 1414 imagedata can be loaded on to the computer system 1400 and subject toprocessing, e.g., post capture processing. The images may be stored inthe data/information portion 1420 of memory 1408 for processing. Theassembly of modules 1418 includes one or more modules or routines which,when executed by the processor 1410, control the computer system toimplement one or more of the image processing operations described inthe present application. The output of multiple optical receiver chainscan be, and in some embodiments is, combined to generate one or moreimages. The resulting images are stored in the data portion of thememory 1408 prior to being output via the network interface 1414, thoughanother interface, or displayed on the display 1402. Thus, via thedisplay 1402 a user can view image data corresponding to one or moreindividual optical chain modules as well as the result, e.g., image,generated by combining the images captured by one or optical chainmodules.

FIG. 15 illustrates a frontal view of an apparatus implemented inaccordance with one embodiment of the present invention whichincorporates multiple optical chain modules. Camera device 1500 includesfour optical chains OCM 1, OCM2, OCM 3 and OCM 4. The outer lens of OCM1, OCM 2, OCM 3 and OCM 4 being shown as circles with a frontal view.OCM1 including a red filter element, OCM 2 including a green filterelement, OCM 3 including a blue filter element. Optical chain module 4passes all three colors and includes a sensor with a multi-color filterelement, e.g., a Bayer filter. The optical chain modules may be the sameas or similar to those previously described in FIGS. 1-3.

FIG. 16 illustrates a frontal view of the outer lenses of an apparatus1605 implemented in accordance with one embodiment of the presentinvention which incorporates multiple optical chain modules and which isdesigned to have little or no gaps between the outer most lenses of thedifferent optical chain modules. The outer most lenses may be theaperture stop lenses in the FIG. 16 embodiment. Apparatus 1605 of FIG.16 includes 7 optical chain modules OCM1, OCM2, OCM3, OCM4, OCM5, OCM6and OCM7 with the outer lens plane corresponding to lenses L1 as viewedfrom the front of the camera device being shown in FIG. 16. The outerlenses L1 of optical chain modules 1, 2, 3, 4, 5, and 6 are positionedso as to surround the outer lens L1 of the optical chain module 7. Theouter lens L1 of the optical chain module 7 being formed in the shape ofa hexagon, i.e., a six sided polygon. The outer lenses L1 of opticalchain modules 1, 2, 3, 4, 5 and 6 being of same shape and size and whencombined with lens L1 of optical module 7 forming a circle. The opticalcenter of each lens L1 of optical chain modules shown as a dark soliddot on the dashed circle. The optical center of lens L1 of optical chainmodule 7 shown as a dot in the center of the hexagon and also in centerof the dashed line. A block separator or other light block may be usedbetween the lenses to stop light leakage between the different lenses.The dots in FIGS. 16 and 17 represent the optical center of theindividual lenses. In some embodiments each outermost lens is a roundconvex lens with its parameter cut to the shape shown in FIG. 16 so thatthe lenses fight closely together. The little or no gap between thefront lenses, e.g., the total area of the gap between the lensesoccupies less than 5% of the total area of the front area of the lensassembly, e.g., circle shown in FIG. 16, occupied by the lenses whenassembled together. The lack of or small size of the gaps facilitatesgenerating combined images with a desirable bokehs or blurs in thecombined image with regard to image portions which are out of focus,e.g., in some cases without the need for extensive and potentiallycomplex processing to generate the combined image.

FIG. 17 illustrates a frontal view of the outer lenses of an apparatus1705 implemented in accordance with one embodiment of the presentinvention which incorporates multiple optical chain modules and outerlenses, e.g., the aperture stop lens for each of the correspondingoptical chains, arranged to have non-uniform spacing between the opticalcenters of the lenses. Thus the FIG. 17 embodiment is similar to theFIG. 16 embodiment but with non-uniform spacing of the optical centersof lenses along the outer parameter of the lens assembly. Thenon-uniform spacing facilitates depth of field determinationsparticularly when performing block processing and the entire field ofview may not be under consideration when processing a block orsub-portion of the captured field of view. The optical chain modulesshown in FIGS. 16 and 17 are the same or similar to those previouslydescribed with reference to FIG. 3 but differ in terms of lens shape,size and/or configuration.

FIG. 18 illustrates another exemplary camera device 1801 including aplurality of first through fifth optical chain modules each of whichincludes an outer lens 1813, 1818, 1817, 1819, or 1821 represented as acircle on the outer lens platter 1803. Rather than being circularlenses, elements 1813, 1818, 1817, 1819, or 1821 could be simple emptyapertures that let in light or might be a clear protective surfacethrough which light can pass such as flat glass, plastic, or a filter.Elements 1813, 1818, 1817, 1819, or 1821 may be followed by a foldelement such as a mirror, which is then followed by lenses, filters,etc. Filter and/or lens position may vary depending on the embodiment.

In both FIGS. 18 and 19 arrows made of dashed lines represent the pathof light for the corresponding optical chain module after light whichentered the outer lens along the optical axis of the outer lens isredirected by the mirror or other light redirection device. Thus, thearrows represents the direction and general light path towards thesensor of the optical chain to which the arrow corresponds.

In the FIG. 18 embodiment each of the optical chain modules includes, inaddition to an outer lens 1813, 1815, 1817, 1819, or 1821, a mirror orother device, e.g., prism, 1823, 1825, 1827, 1829, or 1831 for changingthe angle of light received via the corresponding outer lens.Additionally, as in some of the previously described embodiments such asthe FIGS. 1A, 1B, 1C and 3 embodiments, each camera module includes afilter 1833, 1835, 1837, 1839, or 1841 and an inner lens 1843, 1845,1847, 1849, or 1851. In addition each optical chain module includes asensor 1853, 1855, 1857, 1859, or 1861. For example, the first opticalchain module include outer lens 1813, mirror 1823, filter 1833, innerlens 1843 and sensor 1853.

Filters 1833, 1835, 1837, 1839, or 1841 are mounted on a movablecylinder 1875 represented as a circle shown using small dashed lines.The cylinder 1875 may be rotated and/or moved forward or backwardallowing lenses and/or filters on the cylinder to be easily replacedwith other lenses, filter, or holes mounted on the cylinder 1875. Whilein the FIG. 18 example, an exit hole is provided to allow light to exitcylinder 1875 after passing through one of the filters 1833, 1835, 1837,1839, or 1841 it should be appreciated that rather than an exit holeanother lens or filter may be mounted on the cylinder 1875 allowing twoopportunities for the light to be filtered and/or passed through a lensas is passes through the cylinder 1875. Thus, in at least someembodiments a second filter or lens which is not shown in FIG. 18 forsimplicity is included at the exit point for the light as it passesthrough cylinder 1804. Inner lenses are mounted on cylinder 1885 whichis actually closer to the outside sidewalls of the camera device 1801than the filters mounted on cylinder 1875. Given the large diameter ofmovable cylinder 1885 and the relatively small diameter of the lightbeam as it nears the sensor, it should be appreciated that a largenumber of alternative filters, lenses and/or holes can be mounded oncylinder 1885. As with cylinder 1875 the light can be filtered and/orprocessed by a lens as it enters and leaves cylinder 1885 prior toreaching the sensor of the corresponding optical chain.

In some embodiments lenses mounted on a moveable platter positionedbetween the outer lens platter 1803 and mirrors which may, and in someembodiments are, also mounted on a platter are used to supportautofocus. In such an embodiment the lens platter between the outer lensplatter and mirror platter is moved in or out to perform focusoperations for each of the optical chain modules in parallel. In anotherembodiment, different sets of lens are mounted on the drum 1885 or 1875with different lens sets being mounted with a different offset distancefrom the surface of the drum. By switching between the different sets oflenses by rotating the drum on which the different lens sets aremounted, focusing between different predetermined focus set points can,and in some embodiments is achieved, by simply rotating the drum onwhich the lens sets, corresponding to the different focal distance setpoints, are mounted.

Notably, the FIG. 18 embodiment, by changing the direction of lightthrough the use of mirrors, prisms and/or other devices allows for thelength of the individual optical chains to be longer than the cameradevice is thick. That is, the side to side length of the camera device1801 can be used in combination with a portion of the front to backlength to create optical chains having a length longer than the depth ofthe camera device 1801. The longer optical chain length allows for morelenses and/or filters to be used as compared to what may be possiblewith shorter optical chain lengths. Furthermore, the change in thedirection of light allows for the use of cylinders for mounting lenses,filters and/or holes which can be easily interchanged by a simplerotation or axial, e.g., front to back movement, of the cylinder onwhich the lenses, filters and/or holes corresponding to multiple opticalchains are mounted.

In the FIG. 17 embodiment sensors may be fixed and/or mounted on amovable cylinder. Thus, not only can the lenses, filters and/or holes beeasily switched, changes between sensors or sets of sensor can be easilymade by rotating the cylinder on which the sensors are mounted. While asingle mirror is shown in FIG. 18 in each optical chain module,additional mirrors may be used to further extend the length of theoptical path by reflecting in yet another direction within the housingof the camera device 1801.

It should be appreciated that the FIG. 18 embodiment allows for acombination of lens, filter, and/or hole mounting platters arrangedparallel with the platter extending left to right within the cameradevice and cylinders arranged so that the top and bottom of the cylinderextend in the front to back direction with respect to the camera body,e.g., with the front of the camera being shown in FIG. 18. Cylinders maybe mounted inside of one another providing a large number ofopportunities to mount lens, filters and/or holes along the opticalpaths of each optical chain module and allowing for a large number ofpossible filter/lens/sensor combinations to be supported, e.g., byallowing for different combinations of cylinder positions for differentmodes of operation.

While changing sensors mounted on a cylinder can be achieved by rotatinga cylinder, in the earlier embodiments in which sensors may be mountedon platters, sensors may be changed by rotating or otherwise moving aplatter on which the sensors are mounted.

Note that in the FIG. 18 embodiment the outer lenses of the opticalchain modules are mounted near the center of the front of the cameradevice 1801 as shown, e.g., forming a generally circular pattern ofouter lenses 1813, 1815, 1817, 1819, 1821.

In some embodiments, rather than lenses elements 1813, 1815, 1817, 1819,1821 could be implemented as apertures, e.g., openings, that let inlight. Alternatively, flat glass or another material could be used aselements 1813, 1815, 1817, 1819, 1821 to keep direct out of the camera.The lens, aperture, or clear material 1813, 1815, 1817, 1819, 1821 maybe followed by a fold element or other light redirection element such asa mirror, which is then followed by lenses, filters, etc. In somesystems the ordering in an optical chain is: hole or protective cover,mirror, lens or lenses, filter and then sensor. However, otherconfigurations are possible. For example the filter position can beplaced at a variety of locations depending on the implementation,including right up front before or after the mirror or light redirectiondevice or between lenses.

FIG. 19 is similar to the FIG. 18 embodiment in that it illustratesanother camera device 1901 including a plurality of optical chainmodules which include mirrors or another device for changing the angleof light entering the optical chain module and thereby allowing at leasta portion of the optical chain module to extend in a direction, e.g., aperpendicular direction, which is not a straight front to back directionwith respect to the camera device. FIG. 19 differs from the FIG. 18embodiment in that the outer lenses of the first through fifth opticalchain modules are positioned near the perimeter of the face of thecamera device 1901. This allows for the length of the optical chainmodule to be longer than the length of the optical chains shown in FIG.18. FIG. 19 shows outer and inner cylinders, also some times referred toas drums, 1975, 1985, upon which filters, lenses and holes can and invarious embodiments are mounted as discussed with regard to the FIG. 18embodiment. Thus cylinders 1975 and 1985 server the same or similarpurpose served by cylinders 1875, 1885, respectively. It should beappreciated that in some embodiments the FIG. 19 embodiment includesfilters and lenses mounted on the inner and outer cylinders in the sameor similar manner as filters and lenses are mounted on the cylinders1875, 1885 shown in FIG. 18.

Elements of the FIG. 19 embodiment which are the same or similar to theelements of the FIG. 18 embodiment are identified beginning with “19”instead of “18” and for the sake of brevity will not be described againin detail. For example element 1961 is used to refer to the sensor forthe optical chain module which includes outer lens 1921, mirror/lightredirection device 1931, filter 1941 and inner lens 1951. The cylinder1975 is used to mount the filters while cylinder 1985 is used to mountthe inner lenses.

The camera devices 1801 and 1901 may, and in some embodiments do,include a processor, display and/or other components of the cameradevice shown in FIG. 1A but such elements are not explicitly shown inthe FIGS. 18 and 19 embodiments to avoid complicating the figures andbeing repetitive.

Various functions of the present invention may be and are implemented asmodules in some embodiments. The assembly of modules 1700 shown in FIG.13 illustrates an exemplary assembly of modules, e.g., software orhardware modules, that may be and are used for performing variousfunctions of a image processing system or apparatus used to processimages in accordance with embodiments of the present invention. When themodules identified in FIG. 13 are implemented as software modules theymay be, and in some embodiments of the present invention are, stored inmemory 108 of FIG. 1A in the section of memory identified as assembly ofmodules 118. These modules may be implemented instead as hardwaremodules, e.g., circuits.

The ideas and concepts described with regard to various embodiments suchas those shown in FIG. 19 can be extended so that the input sensors canbe located in a plane, e.g., at the back of the camera device and/or atthe front of the camera device. In some such embodiments the sensors ofmultiple optical chains are mounted on a flat printed circuit board orbackplane device. The printed circuit board, e.g. backplane, can bemounted or coupled to horizontal or vertical actuators which can bemoved in response to detected camera motion, e.g., as part of a shakecompensation process which will be discussed further below. In some suchembodiments, pairs of light diverting devices, e.g., mirrors, are usedto direct the light so that at least a portion of each optical chainextends perpendicular or generally perpendicular to the input and/orsensor plane. Such embodiments allow for relatively long optical pathswhich take advantage of the width of the camera by using mirrors orother light diverting devices to alter the path of light passing throughan optical chain so that at least a portion of the light path extends ina direction perpendicular or generally perpendicular to the front of thecamera device. The use of mirrors or other light diverting devicesallows the sensors to be located on a plane at the rear or front of thecamera device as will now be discussed in detail.

In the FIGS. 20 and 21 embodiments two or more deflection elements areused in each optical chain. Mirrors are exemplary deflection elementsthat may and sometimes are used in the FIGS. 20 and 21 embodiments.Prisms are examples of deflection elements which may, and in someembodiments are, used in place of mirrors. Thus, at least in someembodiments each optical chain includes multiple deflection elements inthe form of mirrors. Deflection elements alter the path of light and,while implemented as mirrors and prisms in some embodiments, may also beimplemented using other devices, e.g., optic elements, which can alterthe path of light. The deflection element may be implemented as a foldedoptic element, folding element, beam guiding element, reflector, waveguiding element. Thus, it should be appreciated that a deflectionelement can take a wider variety of forms and may be implemented as amirror, beam splitter, prism, that can, and in some embodiments is, usedto bend or fold the optical path of an optical chain or light beampassing through an optical chain. In FIGS. 20 and 21 embodiments twodeflection elements are used in each optical chain. In one particularembodiment the deflection elements are implemented as mirrors which,deflect light 90 degrees or substantially 90. However, the deflectionelements need not be of the same type, for example one deflectionelement may be a mirror and another deflection element may be a prism.Furthermore while 90 degrees is used in some embodiments other amountsof deflection may be used in other embodiments.

FIG. 20 illustrates an exemplary diagram of a camera device 2000implemented in accordance with one exemplary embodiment of theinvention. The FIG. 20 diagram is intended for explanation purposes tofacilitate an understanding of various features and thus is not aprecise view of the camera device as perceived from the top but afunctional diagram of the elements from a top view perspective which isintended to convey various aspects of the optical chain configurationsused in the device 2000. The top portion of FIG. 20 corresponds to thefront of the camera device 2000 while the bottom portion corresponds tothe back of the camera device 2000. The body 2001 of the camera extendsfrom left to right with the lens and/or openings 2002, 2004, 20006corresponding to multiple optical chains being mounted in front of thecamera device 2000. A LCD or other display (not shown) may and in someembodiments is, located at the rear of the camera device 2000.

In the camera device 2000 includes a plurality of lens or openings L1through LZ (L1 2002, L2 2004, . . . , LZ 2006) each corresponding to adifferent one of Z optical chains. Note that in FIG. 20 the lenses 2002,2004, . . . , 2006 are loaded in a plane represented by dashed line 2012which extends down towards the bottom of the camera device 2000 which isnot visible in the FIG. 20 diagram. The lenses 2002, 2004, . . . , 2006may be arranged in a circular or other pattern on the front of thecamera device 2000. Each optical chain in the FIG. 20 embodimentincludes multiple mirrors or other light redirecting devices, e.g.,prisms, and a sensor positioned at the end of the optical chain. Forexample, optical chain 1 includes lens L1 2002, first deflection element2022, e.g., a mirror or prism, second deflection element 2024, e.g., amirror or prism, and sensor 1 2038. Optical chain 3 includes lens L22004, first deflection element 2023, e.g., a mirror or prism, seconddeflection element 2025, e.g., a mirror or prism, and sensor 2 2037.Optical chain Z includes lens LZ 2006, first deflection element 2028,e.g., a mirror or prism, second deflection element 2026, e.g., a mirroror prism, and sensor Z 2034. It should be appreciated that deflectionelements, e.g., mirrors and/or prisms, of the first and second opticalchains are located around the cylinder 2020 on which one or more lensesor filters may be mounted as discussed with regard to the otherembodiments. The deflection elements, e.g., mirrors and/or prisms, maybe arranged in a plane positioned parallel to the input plane 2012 withthe light of the different optical chains passing each other, e.g.,crossing, within the cylinder 2020. While a single cylinder 2020 isshown in FIG. 20, multiple cylinders, lenses and/or filters may, and insome embodiments are, used as discussed with regard to the otherembodiments. Note that in the FIG. 20 embodiment the deflectionelements, e.g., mirrors and/or prisms, (2022, 2024), (2023, 2025),(2028, 2026) redirect the light passing through the optical chain towhich the deflection elements, e.g., mirrors and/or prisms, correspondso that at least a portion of the optical path of the optical chainextends perpendicular or generally perpendicular to the input directionin which the input lenses L1, L2, LZ face and parallel to the inputplane 2012. The input plane may be implemented as a mounting device,e.g., circuit board, upon one or more input lenses or openings L1, L2,LZ are mounted or included in. This allows the optical chain to takeadvantage of the left to right width of a camera permitting an overalloptical chain length than would be possible if the optical chain waslimited to the front to back depth of camera device 2000. This allowsfor thin cameras with relatively long optical chains. Notably, the useof two 45 degree mirrors in an optical chain, e.g., mirror 2022 andmirror 2024 in optical chain 1; mirror 2023 and mirror 2025 in opticalchain 2; and mirror 2028 and mirror 2026 in optical chain Z, allows thesensors (2038, 2037, 2034) of the optical chains to be mounted in abackplane 2030 with the sensors being arranged on the backplane 2030 ina plane, as indicated by dashed line 2039, which is parallel to theinput plane 2012. In some embodiments, there are different sensor planescorresponding to different optical chains, and the sensor planes of thedifferent optical chains are parallel to one another. The ability tomount the sensors (2038, 2037, . . . , 2034) on a single backplane 2030allows for the simple movement of the sensors as an assembly maintainingthe relative position of the sensors (2034, 2037, 2038) to one anotheron the backplane 2030 even if the backplane is moved. The cylinder anddeflection elements, e.g., mirrors, may, but need not be, mounted in amanner so that they will move with the backplane 2030 maintaining thealignment of the optical chains to one another as the backplane 2030 ismoved, e.g., up or down or left to right in the camera body 2000. Thus,in some embodiments the backplane 2030 and sensors (2034, 2037, 2038)can be moved in unison, e.g., by applying a force to the backplane 2030to induce motion as may be desired. In some embodiments, the backplane2032 is a moveable plate. In some such embodiments, the plate isarranged parallel to the sensor plane 2039.

In one embodiment, motion sensors 2040 are included in the camera device2000. The motion sensors 2040 may be accelerometers and/or gyroscopesused to detect motion along one or more axis of the camera. In oneparticular embodiment a shake compensation module 2042 is included inthe camera device 2000. The shake compensation module 2042 receivesoutput from the motion sensors 2040 and detects camera movement, e.g.,movement indicative of un-intentional shaking as is common in the caseof hand held cameras. The shake compensation control module 2042 iscoupled to a horizontal actuator 2032 and a vertical actuator 2036 whichare in contact with the backplane 2030 which may be a circuit board. Insome embodiments, the vertical actuator 2036 and horizontal actuator2032 are included as part of a two axis positioning module 2031. In somesuch embodiments, the shake compensation control module 2042 is acontrol module for controlling the two axis positioning module 2031 as afunction of camera movement. The vertical actuator 2036 is shown indashed lines since it is positioned below backplane 2030 and would notbe visible from the top. The vertical actuator 2036 can be used to movethe backplane 2030, e.g. circuit board, up or down while actuator 2032can be used to move the backplane 2030 left or right. In at least oneembodiment backplane 2030 is mounted in a manner that allows motion leftand right, up and down, but which maintains its parallel relationship tothe input plane 2012. In some embodiments backplane 2030 is mounted in aslot which is part of the housing of the camera device 2000. Theactuators 2032, 3036 may be motorized or implemented using elementswhich expand or contract when a voltage is supplied. The shakecompensation control module 2042 controls the supply of power and/orcontrol signals to actuators 2032, 2036 which induces motion of thebackplane 2030 and sensors mounted thereon which is intended tocounteract the shaking. The motion of the backplane 2030 is normally notdetectable to the holder of the camera but can reduce the distorting inthe captured images induced by shaking of the camera housing in whichthe various elements of the camera are mounted. The lenses and/oropenings 2002, 2004, 2006 may not distort or focus the incoming lightand may remain fixed while one or more of the other elements of theoptical chains move, e.g., to compensate for shaking and/or changes thelenses on the cylinder or drum 2020 through which light will pass.

The FIG. 20 embodiment is particular well suited for embodiments whereit is desirable from a manufacturing standpoint and/or shakecompensation standpoint to mount the sensors 2034, 2037, . . . , 2038 onbackplanes such as printed circuit boards or other relatively flatmounting devices whether they be out of metal, plastic, another materialor a combination of materials. In various embodiments, shakecompensation control module 2042 is an image stabilization module fordetermining the amount of movement of the backplane 2030, e.g., amoveable plate, to perform an image stabilization operation tocompensate for camera movement.

It may be observed that the light passing through a first optical chainmodule, e.g., optical chain module 1, in the plurality of optical chainmodules (optical chain module 1, optical chain module 2, . . . , opticalchain module Z), crosses a light path of another optical chain module(optical chain module Z), when the light passing through the firstoptical chain module travels from the first deflection element 2022 tothe second deflection element 2024 of the first optical chain module. Invarious embodiments, at least one of the first and second deflectionelements is a prism. In various embodiments, light beams passing throughdifferent optical chains cross each other when traveling between firstand second deflector elements of their respective optical chain modules.

It should be appreciated that the camera device 2000, as well as thecamera device 2100 shown in FIG. 21 may include the elements of thecamera device 100 shown in FIG. 1A in addition to those shown in FIGS.20 and 21 but that such elements are omitted to facilitate anunderstanding of the elements and configuration which is explained usingFIGS. 20 and 21.

FIG. 21 illustrates an additional exemplary camera device 2100 in whichmirrors (2122, 2124), (2128, 2126) and/or other light redirectingelements are used to alter the path of light in the optical chains sothat the input light input lenses and/or opens can be arranged in one ormore planes at the front of the camera where the lens and/or openingsthrough which light enters the optical chains are also located. Elementsin FIG. 21 which are the same or similar the elements of FIG. 20 arenumbered using similar numbers but starting with the first two digits 21instead of 20. Such similar elements will not be described again expectto point out some of the differences between the FIG. 21 and FIG. 20configurations.

One of the important differences between the devices 2100 and 2000 isthat in the camera device 2100 both the sensors 2134, 2138 and externallenses/openings of the optical chains are located in the front of thecamera. This is made possible by having the second mirror 2124 or 2126direct light to the front of the camera rather than the back of thecamera. In the FIG. 21 embodiment the input plane and the sensor planemay be the same plane or positioned in close proximity to each other. Asin the case of the FIG. 20 embodiment vertical and horizontal actuators2132, 2136 may be provided and used to mechanically compensate fordetected camera shaking.

The FIG. 21 embodiment may be desirable where a manufacturer may want tocombine the input plane assembly and sensor plane assembly into a singleunit as part of the manufacturing processor prior to combining it withthe cylinder/lens assembly 2120.

Numerous variations on the designs shown in FIGS. 20 and 21 arepossible. Significantly, the methods and apparatus of the presentinvention allow for sensors to be arranged parallel to or on anyinternal wall of a camera device while still allowing for a cameradevice to include multiple optical chains in a relatively thin camera.By configuring the sensors parallel to the front or rear walls of thecamera rather than the side walls, the sensors and/or lens can be spreadout and occupy a greater surface area than might be possible if thecamera sensors were restricted to the sidewalls or some otherarrangement.

Notably many of the embodiments are well suited for allowing a LCD orother display to be placed at the back of the camera facing out withoutthe display panel significantly interfering with the overall length ofthe individual optical chain modules included in the camera.

While the invention has been explained using convex lenses in many ofthe diagrams, it should be appreciated that any of a wide variety ofdifferent types of lenses may be used in the optical chain modulesincluding, e.g., convex, concave, and meniscus lenses. In addition,while lenses and filters have been described as separate elements,lenses and filters may be combined and used. For example, a color lensmay, and in some embodiments is, used to both filter light and alter thelights path. Furthermore, while many of the embodiments have beendescribed with a color filter preceding the image sensor of an opticalchain or as using an image sensor with an integrated color filter, e.g.,a Bayer pattern filter, it should be appreciated that use of colorfilters and/or sensors with color filters is not required and in someembodiments one or more optical chain modules are used which do notinclude a color filter and also do not use a sensor with a color filter.Thus, in some embodiments one or more optical chain modules which sensea wide spectrum of color light are used. Such optical chain modules areparticularly well suited for generating black and white images.

In various embodiments image processing is used to simulate a widevariety of user selectable lens bokehs or blurs in the combined imagewith regard to image portions which are out of focus. Thus, whilemultiple lenses are used to capture the light used to generate acombined image, the image quality is not limited to that of anindividual one of the lenses and a variety of bokehs can be achieveddepending on the particular bokeh desired for the combined image beinggenerated. In some embodiments, multiple combined images with differentsimulated bokehs are generated using post image capture processing withthe user being provided the opportunity to save one or more of thegenerated combined images for subsequent viewing and/or printing. Thus,in at least some embodiments a physical result, e.g., a printed versionof one or more combined images is produced. In many if not all casesimages representing real world objects and/or scenes which were capturedby one or more of the optical chain modules of the camera device used totake the picture are preserved in digital form on a computer readablemedium, e.g., RAM or other memory device and/or stored in the form of aprinted image on paper or on another printable medium.

While explained in the context of still image capture, it should beappreciated that the camera device and optical chain modules of thepresent invention can be used to capture video as well. In someembodiments a video sequence is captured and the user can select anobject in the video sequence, e.g., shown in a frame of a sequence, as afocus area, and then the camera device capture one or more images usingthe optical chain modules. The images may, and in some embodiments are,combined to generate one or more images, e.g., frames. A sequence ofcombined images, e.g., frames may and in some embodiments is generated,e.g., with some or all individual frames corresponding to multipleimages captured at the same time but with different frames correspondingto images captured at different times.

While different optical chain modules are controlled to use differentexposure times in some embodiments to capture different amounts of lightwith the captured images being subsequently combined to produce an imagewith a greater dynamic range than might be achieved using a singleexposure time, the same or similar effects can and in some embodimentsis achieved through the use of different filters on different opticalchains which have the same exposure time. For example, by using the sameexposure time but different filters, the sensors of different opticalchain modules will sense different amounts of light due to the differentfilters which allowing different amounts of light to pass. In one suchembodiment the exposure time of the optical chains is kept the same byat least some filters corresponding to different optical chain modulescorresponding to the same color allow different amounts of light topass. In non-color embodiments neutral filters of different darknesslevels are used in front of sensors which are not color filtered. Insome embodiments the switching to a mode in which filters of differentdarkness levels is achieved by a simple rotation or movement of a filterplatter which moves the desired filters into place in one or moreoptical chain modules. The camera devices of the present inventionsupports multiple modes of operation with switching between panoramicmode in which different areas are captured, e.g., using multiple lensesper area, and a normal mode in which multiple lens pointed samedirection are used to capture the same scene. Different exposure modesand filter modes may also be supported and switched between, e.g., basedon user input.

Numerous additional variations and combinations are possible whileremaining within the scope of the invention.

The techniques of the present invention may be implemented usingsoftware, hardware and/or a combination of software and hardware. Thepresent invention is directed to apparatus, e.g., dedicated cameradevices, cell phones, and/or other devices which include one or morecameras or camera modules. It is also directed to methods, e.g., methodof controlling and/or operating cameras, devices including a camera,camera modules, etc. in accordance with the present invention. Thepresent invention is also directed to machine readable medium, e.g.,ROM, RAM, CDs, hard discs, etc., which include machine readableinstructions for controlling a machine to implement one or more steps inaccordance with the present invention.

In various embodiments devices described herein are implemented usingone or more modules to perform the steps corresponding to one or moremethods of the present invention, for example, control of image captureand/or combining of images. Thus, in some embodiments various featuresof the present invention are implemented using modules. Such modules maybe implemented using software, hardware or a combination of software andhardware. In the case of hardware implementations embodimentsimplemented in hardware may use circuits as part or all of a module.Alternatively, modules may be implemented in hardware as a combinationof one or more circuits and optical elements such as lenses and/or otherhardware elements. Thus in at least some embodiments one or moremodules, and sometimes all modules, are implemented completely inhardware. Many of the above described methods or method steps can beimplemented using machine executable instructions, such as software,included in a machine readable medium such as a memory device, e.g.,RAM, floppy disk, etc. to control a machine, e.g., a camera device orgeneral purpose computer with or without additional hardware, toimplement all or portions of the above described methods, e.g., in oneor more nodes. Accordingly, among other things, the present invention isdirected to a machine-readable medium including machine executableinstructions for causing or controlling a machine, e.g., processor andassociated hardware, to perform e.g., one or more, or all of the stepsof the above-described method(s).

While described in the context of an cameras, at least some of themethods and apparatus of the present invention, are applicable to a widerange of image captures systems including tablet and cell phone deviceswhich support or provide image capture functionality.

Images captured by the camera devices described herein may be real worldimages useful for documenting conditions on a construction site, at anaccident and/or for preserving personal information whether beinformation about the condition of a house or vehicle.

Captured images and/or composite images maybe and sometimes aredisplayed on the camera device or sent to a printer for printing as aphoto or permanent document which can be maintained in a file as part ofa personal or business record.

Cameras implemented in some embodiments have optical chains which do notextend out beyond the front of the camera during use and which areimplemented as portable handheld cameras or devices including cameras.Such devices may and in some embodiments do have a relatively flat frontwith the outermost lens or clear optical chain covering being fixed.However, in other embodiments lenses and/or other elements of an opticalchain may, and sometimes do, extend beyond the face of the cameradevice.

Numerous additional variations on the methods and apparatus of thepresent invention described above will be apparent to those skilled inthe art in view of the above description of the invention. Suchvariations are to be considered within the scope of the invention. Invarious embodiments the camera devices are implemented as digitalcameras, video cameras, notebook computers, personal data assistants(PDAs), or other portable devices including receiver/transmittercircuits and logic and/or routines, for implementing the methods of thepresent invention and/or for transiting captured images or generatedcomposite images to other devices for storage or display.

Numerous additional embodiments are possible while staying within thescope of the above discussed features.

What is claimed:
 1. A camera device, comprising: a plurality of opticalchains, a first optical chain in said plurality of optical chainsincluding a lens, a plurality of deflection elements, and a sensor, saidplurality of deflection elements including at least a first deflectionelement and a second deflection element; and wherein light passingthrough the first optical chain in said plurality of optical chainscrosses a light path of another optical chain in said plurality ofoptical chains when the light passing though the first optical chaintravels from the first deflection element to the second deflectionelement of the first optical chain.
 2. The camera device of claim 1,wherein the sensor of each optical chain is mounted in a sensor plane;and wherein the sensor planes of the different optical chains are thesame plane.
 3. The camera device of claim 1, wherein the sensors of thedifferent optical chains are mounted on a movable plate, said movableplate being arranged parallel to said sensor plane.
 4. The camera deviceof claim 3, further comprising: a two axis positioning module coupled tosaid movable plate; and a control module for controlling said two axispositioning module as a function of camera device movement.
 5. Thecamera device of claim 4, wherein said control module is an imagestabilization module for determining the amount of movement of saidmovable plate to perform an image stabilization operation to compensatefor camera device movement.
 6. The camera device of claim 1, whereinmultiple optical chains in said plurality of optical chains each includeat least two mirrors.
 7. The camera device of claim 1, wherein at leastone of the first and second deflection elements is a prism.
 8. Thecamera device of claim 1, wherein light beams passing though differentoptical chains cross each other when traveling between the first andsecond deflection elements of their respective optical chains.
 9. Acamera device, comprising: a plurality of optical chains, a firstoptical chain in said plurality of optical chains including a lens, aplurality of deflection elements, and a sensor, said plurality ofdeflection elements including at least a first deflection element and asecond deflection element; and wherein light passing through the firstoptical chain in said plurality of optical chains crosses a light pathof another optical chain when the light passing though the first opticalchain travels from the first deflection element to the second deflectionelement of the first optical chain.
 10. The camera device of claim 9,wherein light beams passing though different optical chains cross eachother when traveling between the first and second deflection elements oftheir respective optical chains.
 11. The camera device of claim 9further comprising an additional optical chain which includes an imagedeflection element configured to change the direction of optical rayspassing along an optical axis of a lens by substantially 90 degrees todirect said optical rays passing along said optical axis onto a sensorof the additional optical chain in which said image deflection elementand said lens are located.
 12. The camera device of claim 9, where saidplurality of optical chains includes at least three optical chains. 13.The camera device of claim 12, wherein the sensor of each optical chainin said plurality of optical chains is mounted in a sensor plane, thesensor plane of different optical chains being parallel to one another.14. The camera device of claim 9, wherein the sensors included in saidplurality of optical chains are mounted on a plate.
 15. The cameradevice of claim 14, wherein said plate is a movable plate.